Welcome, Guest. Please login or register.
Amiga Kit Amiga Store Iridium Banner AMIStore App Store A600 Memory

AuthorTopic: THOR's Shell Hacks  (Read 8650 times)

0 Members and 2 Guests are viewing this topic.

Offline nyteschayde

  • VIP / Donor - Lifetime Member
  • Hero Member
  • *****
  • Join Date: Mar 2002
  • Posts: 636
  • Total likes: 0
    • http://www.nyteshade.com
Re: THORs Shell Hacks
« Reply #15 on: February 05, 2015, 04:02:20 AM »
Thomas I just want to say, that you seem to be a treasure trove of really useful information for the Amiga. The set interactive bit is fantastic. I've done a lot of Amiga shell scripting and used to have a whole directory dedicated to command line scripts that I'd built.

This stuff is great. If you don't have a blog, I would suggest that you put one up. Especially when it comes to coding. There are fewer and fewer resources every year when it comes to doing more than opening a window and having it shut after 2 or 3 seconds.

If you do modern app development on MacOS, Windows or Linux, making comparisons to those platforms on how things are done could be very valuable. Even comparing to the web if you've done web dev.

I do mostly web dev and iOS work these days but usually want to jump in and start but don't know how. I recently asked you for an example on how to load an image using datatypes. I'm having a few issues with my compiler but otherwise it's fantastic

I'd love to know the equivalent of rendering a button and attaching a method that gets fired when the button is pressed. And after I get the rest of the image stuff under control I'll poke around your website for clues there.

I just want to say that your input is greatly appreciated.
Senior MTS Software Engineer with PayPal
Amigas: A1200T 060/603e PPC • A1200T 060 • A4000D 040 • A3000 (x2) • A2000 Vamp/V2 • A1200 (x4) • A1000 (x3) • A600 Vamp/V1 • A500
 

Offline Thomas Richter

Re: THORs Shell Hacks
« Reply #16 on: February 05, 2015, 05:56:38 AM »
Quote from: ChaosLord;782958
Is there any way when writing scripts and using the shell to use a logical replacment for *, such as CONSOLE:  ?

This may or may not be identical to *, there may be subtle differences. More on this later.
 

Offline gertsy

  • Lifetime Member
  • Hero Member
  • *****
  • Join Date: May 2006
  • Posts: 2296
  • Country: au
  • Total likes: 1
    • http://www.members.optusnet.com.au/~gbakker64/
Re: THORs Shell Hacks
« Reply #17 on: February 05, 2015, 07:26:31 AM »
Thomas. Great stuff. These insights should be in a Blog.?
 

Offline Thomas Richter

THOR's Skip Lab
« Reply #18 on: February 05, 2015, 10:00:31 AM »
Today, I'll try to cover a bit a dark edge of the Amiga Shell, and these are the two commands "SKIP" and "LAB", and why they sometimes didn't work before 3.9. Of the two, LAB is actually not a command at all - or rather,  it is, but it doesn't do anything, while still being necessary.

Flow control - that's a key feature of every programming language. That is, to control where program flow goes. "if-then", "for-next", "do-while", "repeat-until", this kind of stuff. Unfortunately, the only thing AmigaOs has is "if-then", and "goto", even though the latter is pronounced "skip". It's the same ugly command that has been banned from many more reasonable languages for good reason - it creates spagetti code.

Anyhow, it's all we got, so here is how it works: If you have a "lab foo" somewhere in a shell script, then a "skip foo" continues control to this very position in the script. Simple enough? Not quite. If the label is *above* the skip command, i.e. on a line "higher up in the code", you need to say "skip BACK foo".

Simple enough? Not quite. If there is an "execte" somewhere between the label, or you are calling a shell script by some other means - probably because you have a file whose 's' bit is set, it doesn't work at all. Or at least, it did not *used* to work before Os 3.9 latest BoingBag.

Why now does the freaking shell care about execute or not? That's again one of the sad stories of "bad design decision". Remember, commands have an input stream, and an output stream, you can redirect with "<" or ">". In  a sense, the shell has the same: It has an interactive input, which is the console the user types on and the shell connects the input of commands it executes to. This is the console. This stream is used for the "ASK" command, for example, to read user input. But the shell has *also* a command input, and this is where the commands for the shell come from. That's usually also the console, as the shell executes the commands as you enter them.

Now, however, if you execute a script (explicitly, or implicitly by a set s-bit), the shell does the following: It connects its "command input" to the file of the script. Sounds like the easiest thing to do: Magically, the input comes now from the script file. Since the interactive input is still connected to the console, the "ASK" command and all other commands still receive that as standard input, and hence read their input from the console as they should.

That's all pretty lean and neat unless... you want to execute a script from a script. Ooops, what do we do now, the CBM engineers said? They came up with the bloddiest ugly hack you can think of: Take the script to be executed, copy that to the RAM-disk (actually, T:), and append the rest of the script that is currently executed (and hence, the script that calls the other script) and append that at the bottom, then execute the new script.

See where I'm getting? Since the top of the current script is lost, and hence any "LAB"  command is lost, a "skip back" to a label that is below any execute command cannot see the "LAB" anymore, and hence, cannot skip back.

Luckely, the 3.9 shell is smarter, and does what every proper programming language does with recursion: It keeps a stack. Thus, yes, you can "skip back". In case you care, the stack is kept in some shell variables (you can see them with the "set" command, to be discussed in a later edition).

What the 3.9 shell does not is to "get away" with skip, and introduce *useful* flow control, as in "for" or "while". It would not be overly hard to do, but simply was not done. A bit of tradition still had to remain, I'd say.
 

Offline Duce

  • Off to greener pastures
  • Hero Member
  • *****
  • Join Date: Jul 2009
  • Posts: 1699
  • Total likes: 0
    • http://amigabbs.blogspot.com/
Re: THORs Shell Hacks
« Reply #19 on: February 05, 2015, 04:00:28 PM »
These are very helpful, thanks for sharing and keep 'em coming.
 

Offline Thomas Richter

Re: THOR's Shell Hacks
« Reply #20 on: February 05, 2015, 09:35:04 PM »
Just in case anyone wonders where this "append files" hack is located: That's actually part of the "execute" command. Execute is really source of miracles all over the place. It has a "simple execution path", and an "elaborated execution path". In the simple path, it just modifies the command input of the running shell, letting it grab commands from the script file. However, if the script contains "dot commands", i.e. like ".key blabla" to allow arguments, then it's the "elaborate path". Also, if execute finds that it is itself executed from a script, it runs the elaborated path, which does all the mess with copying and appending files. The latter case is no longer necessary with BB 3.9.
 

Offline Gulliver

Re: THOR's Shell Hacks
« Reply #21 on: February 06, 2015, 12:52:58 AM »
Hi Thomas,
               I have two questions about Shell 45.x:

1) Is it possible for the Shell module to be started with RemLib to avoid reboots?

2) Can this Shell module run in a 68000 Amiga?

Thanks in advance.
 

Offline Thomas Richter

Re: THOR's Shell Hacks
« Reply #22 on: February 06, 2015, 01:00:17 AM »
Quote from: Gulliver;783145
1) Is it possible for the Shell module to be started with RemLib to avoid reboots?

No, but its possible to use the "resident" command in the startup-sequence to replace the shell module, very much like it was done in AmigaOs 1.3. Then, however, you don't have the boot shell replaced. The shell is both a resident module (more on that later) and an exec module.  

Code: [Select]
resident CLI L:Shell-Seg SYSTEM pure
resident Shell L:Shell-Seg SYSTEM pure
resident BootShell L:Shell-Seg System Pure
Quote from: Gulliver;783145
2) Can this Shell module run in a 68000 Amiga?


Yes. Even though I will get a couple of bad comments from the cycle-counter party, there is nothing specific in the shell that would profit from an 68020 or above, so it has bee compiled for plain 68K.
« Last Edit: February 06, 2015, 01:04:47 AM by Thomas Richter »
 

Offline Thomas Richter

Re: THOR's Shell Hacks: ViNCed
« Reply #23 on: February 08, 2015, 10:31:10 AM »
Ok, now let's have a look at the console the shell runs in. CBMs original console, CON:, was really a pretty poor excuse. It couldn't do much except moving the cursor back and right, and offering a line editor without a history. Then, with Kickstart 1.3, NEWCON was introduced which included at least a history.

At the same time, ViNCEd (back then VNC = VeryNewCon) saw the light of the day, and featured a full screen editor allowing you to move the cursor wherever you want, a history and TAB expansion. For the next issues of this little thread, we'll need this console, namely the version that comes with AmigaOs 3.9. There are earlier releases on the Aminet which are somewhat less powerful.

ViNCed has often been critized for being "not exactly like XYZ", though that was pretty much the point - being different. After all, Amiga is different as well.

So anyhow, here are - as a starter - a couple of instructions how to make it "less different". TAB expansions then go to the console window (instead of a requester) and the cursor up/down navigate in the history (instead of moving the cursor). That's pretty much the settings it came with in Os 3.9, though probably people installed it otherwise.

First, start the Shell and click on "Settings" top right in the window border. SetVNC pops up. Now navigate to the shell page:



For that, click on the "shell" tab (item 1 in the above graphics), then press on the "forward arrow" (item 2 in the above graphics) until you see "Shell page 3 (of 7)" in the display.

Then click on "Requester if expansion is ambiguous" (item 3) which will already give you the matching files on the first "TAB", then press again the forward arrow (item 4). This gives you shell page 4:



On this page, select "List expansions on the console" (item 5, see above), which will disable the requester, but will just print the matching files in the console, and - if you like - "Do not match characters behind cursor" (item 6, see above). This will only match the start of the file upfront the cursor, but not the characters behind it.

Let us now look at the cursor keys. For that, click on the "Keyboard" tab



which is item 1 in the above figure. Then, click onto the box "2", and press the cursor up key on the console. The word "Up" appears in the box. This means that you'll now change the definition of the "Up" key. Finally, click on the forward arrow (item 3) to go to the second keyboard page:



From the list (item 4) select "History up", then click on "Accept" (item 5), then go back to the previous page (item 6) with the backwards arrow.

There, click again into the box, and press on the "Down" key, then go forward again, and select "History down".

Finally, click on "Save" to make the changes permanent, "Use" to test them in all open shell windows, or "To Window" just to test them in the current window.

This gives you a "more standard" shell console. More on all this tomorrow.
 

Offline Elwood

Re: THORs Shell Hacks
« Reply #24 on: February 08, 2015, 09:30:24 PM »
Quote from: Thomas Richter;782863
the Os 4.x shell is based on the Os 3.9 shell.


This is wrong. H&P never give any source code from 3.5/3.9.
As a result, in OS4 Utilities/IconEdit is the one from 3.1.
Philippe "Elwood" Ferrucci
AmigaOS 4.x betatester
Amiga Translator Organisation
My Homepage......
 

Offline kolla

Re: THORs Shell Hacks
« Reply #25 on: February 09, 2015, 05:25:36 AM »
Hah, this means popcorn!
B5D6A1D019D5D45BCC56F4782AC220D8B3E2A6CC
 

Offline Thomas Richter

Re: THORs Shell Hacks
« Reply #26 on: February 09, 2015, 07:19:38 AM »
Quote from: Elwood;783328
This is wrong. H&P never give any source code from 3.5/3.9.
As a result, in OS4 Utilities/IconEdit is the one from 3.1.

No, H&P didn't, but I did. Remember, the 3.9 shell is not by H&P and was only contributed to H&P under a separate contract. The reason why IconEdit is the one from 3.1 is because H&P did not want to contribute reaction.  Thus, yes, the Os 4.0 shell is pretty much the one from Os 3.9. I don't know how many fixes I made later on where integrated into the 4.x release, but I made sure that Olsen always had access to my code.
 

Offline olsen

Re: THORs Shell Hacks
« Reply #27 on: February 09, 2015, 01:26:43 PM »
Quote from: Thomas Richter;783373
No, H&P didn't, but I did. Remember, the 3.9 shell is not by H&P and was only contributed to H&P under a separate contract. The reason why IconEdit is the one from 3.1 is because H&P did not want to contribute reaction.  Thus, yes, the Os 4.0 shell is pretty much the one from Os 3.9. I don't know how many fixes I made later on where integrated into the 4.x release, but I made sure that Olsen always had access to my code.
If I remember correctly, we had to start over from scratch for OS4, regarding the components which make up what is commonly lumped together as "the shell".

Those components are 1) the user interface which you use to enter commands, edit command lines, perform copy & paste and which displays the output of commands, and then there is 2) the part which loads and runs these commands. These are Kickstart modules named "con-handler" (1) and "shell" (2).

The major reasons for starting over were in portability (ViNCed was written in 68k assembly language, and would have had to be reimplemented), and the unresolved limitations of the "shell" Kickstart module which, for example, in Kickstart versions 2.04 through 3.1 had a fixed upper limit for the length of an individual command line (the OS4 version has no fixed upper limit).
 

Offline Thomas Richter

Re: THOR's Shell Hacks: CONSOLE jobs.
« Reply #28 on: February 09, 2015, 01:54:47 PM »
Earlier in this thread we had the question whether * could be substituted with CONSOLE:. The answer is "maybe not, it depends". Actually, if you use * or CONSOLE:, the dos.library does quite the same: It checks for the console process registered for the running process, and forwards the request to there.  

What happens in the console is then up to the implementation of the console, and this "depends". For ViNCEd (see above for installation hints), the usage of CONSOLE: is related to job-control. ViNCEd can run multiple shells in the same window, and can switch between them. There is only one in the foreground, and this shell can print to the console window. All others keep in the background, and once they try to print data to the window, they get suspended. The way how ViNCEd can tell the shells apart is by the "console name" (or rather, how ViNCEd calls it, by the "owner").



So let's try this: Open a ViNCEd shell. For my system, the new shell opens as shell process #7, so it prints for me the following as prompt:

Code: [Select]
7.SCSI:>
This is "shell #7", and the directory is SCSI:, which is the name of my hard disk. Let's start a new shell under a new console owner:

Code: [Select]
7.SCSI:> newshell console:myname &
Now, the "&" at the end tells the shell to detach the next command, i.e. this is similar to "run". Equivalently, we could have used

Code: [Select]
7.SCSI:> run newshell console:myname
The result of this command is a message such as following:

Code: [Select]
[CLI 8]  : newshell suspended. [ViNCEd output]
This means: A new shell had been created, shell #8, and the shell had been suspended because it tried to output something, though currently the active owner is shell #7, the one you are typing in.

You can list the active shells with the status command:
Code: [Select]
7.SCSI:> status
This will print the list of all currently active shells, and the commands they are running. Shell #7 runs the status command (clearly), shell #8 is currently idle (suspended) and runs no command.

Let's switch to shell #8 now:
Code: [Select]
7.SCSI:> fg 8
"fg" is a tiny script in s:. If it is not there, you can also type

Code: [Select]
7.SCSI:> setvnc foreground 8
which is just want the script does. It moves shell #8 in foreground, and shell #7 to background.

The prompt now tells you that you are in shell #8:

Code: [Select]
8.SCSI:>
You can place this shell back into background by "fg 7", or simply "bg", which means, "please put the current shell into background".

Code: [Select]
8.SCSI:> bg
We are now back in shell #7. For creating a new shell, ViNCEd has a short-cut. This is the keyboard combination Control+Z. It only works if the current shell is busy, i.e. you are not able to submit a command. What this keyboard combination does internally is to first pick a unique name, and then call "newshell console:xxxx" with "xxxx" replaced by that name, just to give you the chance to enter a command again.

Now, back to the consoles and the console names. Above, we created a new shell with "newshell console:myname". Clearly, any program that wants to print into "console:newname" or uses this as a path name identifies now itself as part of this new shell. "newname" identifies the "console owner", and the name of this owner is "newname". If you write into "console:newname", and this console owner is currently not in foreground, the command is suspended.

Let's try this: First switch back to shell #7, then print something to the owner of console 8, which is not in foreground:
Code: [Select]
8.SCSI:> fg 7
7.SCSI:> echo "hi" >console:myname
as a result, the echo command is suspended. Because it is suspended, the shell does not return, and you are left without a shell in the window.

What can we do? Well, fetch a free one. This is what Control+Z does, so press the control key, hold it, and press Z, then release both keys. You return to a free shell, which is shell 8.
Code: [Select]
8.SCSI:>
If you place this now into background, finally, the echo command is allowed to print.

Now, if "console:" addresses a named console, what does "console:" stand for? It is the "startup console", i.e. the console that was started along with the window. And what is "*" for? It is the "current console", namely the console that is currently active and in foreground.

Thus, "CONSOLE:" and "*" are not necessarily identical, or rather, they are only identical if the current console is the console that was created along with the window - or the only console.

Thus, it is typically safe to output to "*", as this is in foreground always, and it is not always safe to output to "CONSOLE:" as this console might be in background, and you might get suspended.

Thus, "CONSOLE:" != "*", and the file name behind "CONSOLE:" has a specific meaning, it identifies one out of several consoles that share the same window as output. All the job control and output control is done by ViNCEd, and you can switch between the consoles with the SetVNC program or tiny scripts in S:.

There are, of course, corresponding "DOS Packets" that do the switching, i.e. there is no "magic interface", but rather "job control commands" send to the console the same way output is written to the console.
 

Offline Thomas Richter

Re: THORs Shell Hacks
« Reply #29 on: February 09, 2015, 02:06:52 PM »
Quote from: olsen;783396
The major reasons for starting over were in portability (ViNCed was written in 68k assembly language, and would have had to be reimplemented), and the unresolved limitations of the "shell" Kickstart module which, for example, in Kickstart versions 2.04 through 3.1 had a fixed upper limit for the length of an individual command line (the OS4 version has no fixed upper limit).

For reference, the 3.9 version does not have an upper limit either. It uses its own string buffering mechanism and self-expanding buffers, i.e. whenever you use a command longer than the current buffer, a new buffer of the necessary size is created. The 3.1 shell had a maximum command length of 512 bytes.

I wonder what happened with all the other shell bugs (e.g. the one related to "skip back" I mentioned above).