Amiga.org

Amiga computer related discussion => Amiga Tutorials => Topic started by: Thomas Richter on February 01, 2015, 08:27:47 PM

Title: THOR's Shell Hacks
Post by: Thomas Richter on February 01, 2015, 08:27:47 PM
Hi folks,  AmigaOs has two main user interfaces: The workbench and the shell. The workbench uses the mouse, icons, windows, menus... all the neat stuff. So let's talk about the other one: The shell. This is more traditional, uses the keyboard, a single window, and is less intuitive to use.  Apparently - and as it is usually the case - manuals a put aside rarely ever read. Unfortunately, this means that you may miss a lot of the new features of Os 3.9 that were introduced.  This thread, in loose sequence, will present "lesser known features" of the Amiga shell. Unless said otherwise, this thread uses the AmigaOs shell of the latest BoingBag, from Os 3.9, and the console from AmigaOs 3.9, ViNCEd. An older version of ViNCEd can be found in aminet, though in general, you'll need the latest.  All tutorials start with opening a shell window. It's usually top level on the workbench disk, with an icon named "Shell". SYS:Systems also contains an icon, usually named "CLI". That's the same shell, but a less useful user interface that lacks some of the features. We need the former one.  Where exactly these icons are is of course a matter of taste.
Title: Re: THORs Shell Hacks
Post by: Thomas Richter on February 01, 2015, 08:33:52 PM
We'll start today's thread with shell scripts. The shell takes commands, in the form of "command arguments". If you type in "list", in the shell window, that executes the list command.

If you type in "copy s:startup-sequence to ram:" this copies the startup-sequence to the ram-disk. "S:Startup-sequence" is a shell script, a series of commands, one executed by another. This one in particular is important as it is executed by the system for booting up.

One particular "nice" feature makes up today's "Shell Hack": Interactive debugging, and tracing of shell scripts. Tracing means that the shell shows you which command it is about to execute, and you may run the command, skip over it or stop tracing.

To enable tracing, use the following command in the shell:

set interactive on

If this is put top in a shell script, for example into the startup-sequence (with an editor of your choice), the shell will prompt you for each command it is going to be executed. If you press RETURN, the shell will run the command. If you press N or DEL, the shell will skip over it. If you press ESC, the shell will abort tracing and execute the rest of the script without bothering you further. If you press Control+D, the script will be aborted.
Title: Re: THORs Shell Hacks
Post by: DutchinUSA on February 01, 2015, 08:36:34 PM
Thanks, I didn't know the interactive thing, that's cool and helpful ! Keep 'em coming :)
Title: Re: THORs Shell Hacks
Post by: trekiej on February 01, 2015, 08:47:12 PM
Most excellent! (guitar sounds)
Title: Re: THORs Shell Hacks
Post by: kolla on February 02, 2015, 12:50:17 AM
Could you point out when you describe something that specifically requires shell newer than that of OS3.1? Personally I use your shell even on "3.1ish" setups.
Title: Re: THORs Shell Hacks
Post by: Thomas Richter on February 02, 2015, 01:28:59 AM
Quote from: kolla;782812
Could you point out when you describe something that specifically requires shell newer than that of OS3.1?

All of this thread will cover specifically 3.9 features, unless noted otherwise. Thus, "interactive" is new in 3.9 (well, not exactly "new" by any meaning of the word "new" I know, but at least "latest").
Title: Re: THORs Shell Hacks
Post by: kolla on February 02, 2015, 03:48:59 AM
Ok, great!
Title: Re: THORs Shell Hacks
Post by: Oldsmobile_Mike on February 02, 2015, 04:58:31 AM
Great thread idea!  Makes me wish we had a way to make it a "sticky", so would always appear at the top.  Looking forward to reading more of your tips!

:banana::banana::banana:
Title: Re: THORs Shell Hacks
Post by: broadblues on February 02, 2015, 12:10:07 PM
Cool.

Set interactive on

works in the AmigaOS 4.x shell too. I knew about 'set echo on' but not the interactive switch, thanks!
Title: Re: THORs Shell Hacks
Post by: Thomas Richter on February 02, 2015, 09:29:20 PM
Quote from: broadblues;782833
Set interactive on

works in the AmigaOS 4.x shell too. I knew about 'set echo on' but not the interactive switch, thanks!

Naturally, that's because the Os 4.x shell is based on the Os 3.9 shell.
Title: THOR's Shell Hacks (2)
Post by: Thomas Richter on February 02, 2015, 09:47:18 PM
Today's edition is on redirection. It's less useful on AmigaOs than it is on Unix - it allows to direct the output of a command to another file (instead of the console), or to take the input of a command from a file rather than the console.

Everybody knows the classics: >filename redirects the output, filename, which redirects both input and output to the same file, and >>filename, which redirects the output, but instead of overwriting the target file, appends to it.

AmigaOs 3.9 adds more. To understand them, be reminded that there is a third stream, the "error output", which goes usually to the console. Also be reminded that the console spells "*" in AmigaOs (or, actually, Tripos, which is where the dos came from). It is not a "wild card" unlike many other popular osses. Thus, to "redirect" the output to the console, one would write ">*", and to take input from the console, you write "<*". Of course, you usually don't do that simply because that's what the input and output is connected to anyhow.

Now, to redirect the error output, AmigaOs adds *>filename, read as "console goes to file". That's most useful if the target "file" is actually some type of interactive terminal, for example AUX: (the serial console), thus "*>AUX:" makes the console the serial console, and by that also the standard error.

What ">>" is for ">" is "*>>" to "*>": The redirection *>>filename redirects the error output to a file, but instead of overwriting what is there, it appends to the end.

Then, we have "*<>", which redirects the standard error into the same steam the standard output is going to. It does not take an argument. That is, error output goes into the same file the standard output goes to. Thus "command >ram:fool *<>" redirects the error output to the RAM-disk, too.

Last but not least, there is "<<". This redirects standard input, but it's a bit special. It does not take input from a file, but rather from a script a command is placed in. With "<<", part of the script becomes the input, namely everything below the current command line and an "end marker", a special character sequence that is also placed as argument behind "<<".

To give an example, consider the following shell script:

echo "script starts here"
command <foobar
END
echo "script ends here"

redirects the input of "command" to the same script, and everything up to "END" is the input to the command, and the line "foobar" is the literal input to the file. That's most useful if you have to provde complex inputs to a command, but you don't want to create a temporary file in RAM: Instead, the shell does that for you.

If you know the Linux "bash" shell, that's pretty much how it works there, too. On linux "*>" would be spelled "2>", "*>>" would  be "2>>" and "*<>" would be "2>&1", but since the Amiga shell doesn't have "file descriptors" as Linux does, and only has "two and a half" standard descriptors (CIS,COS,CERR) instead of the many Linux has, "2" was a particuarly bad (and unintuitive) choice for the error output. "*>" was picked because it is just the reverse of ">*". The former redirects the console to a file, the latter redirects output to a console, so it's easy to remember.
Title: Re: THOR's Shell Hacks - it's full of stars!
Post by: Thomas Richter on February 03, 2015, 09:16:22 PM
Do you know all the meaning of * in an Amiga Shell? Actually, there are plenty, and it's not always easy to keep them apart.

So, first, if you put the star * *outside of quotes*, it means the current console. So, for example, if I place a

copy foobar to *

it means: Copy the contents of the file onto the screen. Strange, since the star usually means a "wild card" for many other Os'es.

Things change if you put the star in double quotes, however. There, the star is the "escape character" of BCPL, quite like the backslash in C. Since AmigaOs was traditionally Tripos, written in BCPL, it is still this way today.

That is:

echo "*e[H"

puts the cursor in the home position, i.e. top left. This is because "*e" is the escape sequence for the ASCII character "escape" (hex 0x1c, decimal 27), Escape-[ is the "CSI", the "control sequence introducer, and "CSI-H" is the console command, understood by the console window, to put the character into the top-left position.

Thus, the star means something different *in* quotes compared to *outside quotes*. If you use

echo *e[H

instead, then that's just putting the literal string on the console.

Now, in Os 2.x, CBM rewrote the dos.library. Or rather, they simply took "arp", the "Amiga Replacement Project", made a couple of modifications, and used that as a new basis. In arp, a flag exists to change the interpretation of wild-cards, i.e. patterns that match several files to run a job more efficiently. For example,

copy #? to *

copies all files in the current directory to the console, #? matching all files. The expansion of the wild card #? is done by the copy command, not by the shell (unlike in other operating systems, Linux for example, where this expansion is under shell control).

Now we come to the third meaning of the star. Unfortunately, arp included an option in its pattern matching to use * as a replacement for #? "by popular demand". Thus, if a command uses the arp = now dos.library pattern matcher, * *may* (by choice of the user) get another meaning, namely "match every file".

This now becomes really confusing, as you need to remember when a command uses the pattern  matcher (then *=#?) and when not (then *=current console).

So, for example, with the arp-"*" flag set (several patches provide this option),

copy * to *

means "copy all files to the console", where the first "*" is seen by the pattern matcher, so it means #? = all, and the second star is not, so it means console.

Unfortunately, you have to know this, and since you usually cannot know (how, anyhow?) it's usually *not* a good idea to enable this option. There are already two meanings of the star, adding a third one is not a particularly good choice, even more so as it depends on implementation decisions of the shell command and is not under direct control of the user.

Avoid this stuff, and learn that "Amiga is a bit special" and uses #? as wildcard, and not *. * is the BCPL escape sequence within quotes, or the console as file name outside quotes.
Title: Re: THOR's Shell Hacks - it's full of stars!
Post by: Steady on February 03, 2015, 11:38:28 PM
Thanks Thomas,

I always wondered why * as a wildcard didn't really work when enabled. That copy example clarifies it.

Personally, I always disabled the option. #? works just fine for me :-)
Title: Re: THORs Shell Hacks
Post by: ChaosLord on February 04, 2015, 11:34:33 AM
Is there any way when writing scripts and using the shell to use a logical replacment for *, such as CONSOLE:  ?

That would mean I could avoid using * altogether and never get confused by anything ever again.
Title: Re: THORs Shell Hacks
Post by: danwood on February 04, 2015, 12:45:16 PM
Quote from: Thomas Richter;782778


set interactive on


Great tip Thomas, and there was me booting with no startup-sequence and running snoopdos before manually executing SS all these years.  

Really useful.
Title: Re: THORs Shell Hacks
Post by: nyteschayde on February 05, 2015, 04:02:20 AM
Thomas I just want to say, that you seem to be a treasure trove of really useful information for the Amiga. The set interactive bit is fantastic. I've done a lot of Amiga shell scripting and used to have a whole directory dedicated to command line scripts that I'd built.

This stuff is great. If you don't have a blog, I would suggest that you put one up. Especially when it comes to coding. There are fewer and fewer resources every year when it comes to doing more than opening a window and having it shut after 2 or 3 seconds.

If you do modern app development on MacOS, Windows or Linux, making comparisons to those platforms on how things are done could be very valuable. Even comparing to the web if you've done web dev.

I do mostly web dev and iOS work these days but usually want to jump in and start but don't know how. I recently asked you for an example on how to load an image using datatypes. I'm having a few issues with my compiler but otherwise it's fantastic

I'd love to know the equivalent of rendering a button and attaching a method that gets fired when the button is pressed. And after I get the rest of the image stuff under control I'll poke around your website for clues there.

I just want to say that your input is greatly appreciated.
Title: Re: THORs Shell Hacks
Post by: Thomas Richter on February 05, 2015, 05:56:38 AM
Quote from: ChaosLord;782958
Is there any way when writing scripts and using the shell to use a logical replacment for *, such as CONSOLE:  ?

This may or may not be identical to *, there may be subtle differences. More on this later.
Title: Re: THORs Shell Hacks
Post by: gertsy on February 05, 2015, 07:26:31 AM
Thomas. Great stuff. These insights should be in a Blog.?
Title: THOR's Skip Lab
Post by: Thomas Richter on February 05, 2015, 10:00:31 AM
Today, I'll try to cover a bit a dark edge of the Amiga Shell, and these are the two commands "SKIP" and "LAB", and why they sometimes didn't work before 3.9. Of the two, LAB is actually not a command at all - or rather,  it is, but it doesn't do anything, while still being necessary.

Flow control - that's a key feature of every programming language. That is, to control where program flow goes. "if-then", "for-next", "do-while", "repeat-until", this kind of stuff. Unfortunately, the only thing AmigaOs has is "if-then", and "goto", even though the latter is pronounced "skip". It's the same ugly command that has been banned from many more reasonable languages for good reason - it creates spagetti code.

Anyhow, it's all we got, so here is how it works: If you have a "lab foo" somewhere in a shell script, then a "skip foo" continues control to this very position in the script. Simple enough? Not quite. If the label is *above* the skip command, i.e. on a line "higher up in the code", you need to say "skip BACK foo".

Simple enough? Not quite. If there is an "execte" somewhere between the label, or you are calling a shell script by some other means - probably because you have a file whose 's' bit is set, it doesn't work at all. Or at least, it did not *used* to work before Os 3.9 latest BoingBag.

Why now does the freaking shell care about execute or not? That's again one of the sad stories of "bad design decision". Remember, commands have an input stream, and an output stream, you can redirect with "<" or ">". In  a sense, the shell has the same: It has an interactive input, which is the console the user types on and the shell connects the input of commands it executes to. This is the console. This stream is used for the "ASK" command, for example, to read user input. But the shell has *also* a command input, and this is where the commands for the shell come from. That's usually also the console, as the shell executes the commands as you enter them.

Now, however, if you execute a script (explicitly, or implicitly by a set s-bit), the shell does the following: It connects its "command input" to the file of the script. Sounds like the easiest thing to do: Magically, the input comes now from the script file. Since the interactive input is still connected to the console, the "ASK" command and all other commands still receive that as standard input, and hence read their input from the console as they should.

That's all pretty lean and neat unless... you want to execute a script from a script. Ooops, what do we do now, the CBM engineers said? They came up with the bloddiest ugly hack you can think of: Take the script to be executed, copy that to the RAM-disk (actually, T:), and append the rest of the script that is currently executed (and hence, the script that calls the other script) and append that at the bottom, then execute the new script.

See where I'm getting? Since the top of the current script is lost, and hence any "LAB"  command is lost, a "skip back" to a label that is below any execute command cannot see the "LAB" anymore, and hence, cannot skip back.

Luckely, the 3.9 shell is smarter, and does what every proper programming language does with recursion: It keeps a stack. Thus, yes, you can "skip back". In case you care, the stack is kept in some shell variables (you can see them with the "set" command, to be discussed in a later edition).

What the 3.9 shell does not is to "get away" with skip, and introduce *useful* flow control, as in "for" or "while". It would not be overly hard to do, but simply was not done. A bit of tradition still had to remain, I'd say.
Title: Re: THORs Shell Hacks
Post by: Duce on February 05, 2015, 04:00:28 PM
These are very helpful, thanks for sharing and keep 'em coming.
Title: Re: THOR's Shell Hacks
Post by: Thomas Richter on February 05, 2015, 09:35:04 PM
Just in case anyone wonders where this "append files" hack is located: That's actually part of the "execute" command. Execute is really source of miracles all over the place. It has a "simple execution path", and an "elaborated execution path". In the simple path, it just modifies the command input of the running shell, letting it grab commands from the script file. However, if the script contains "dot commands", i.e. like ".key blabla" to allow arguments, then it's the "elaborate path". Also, if execute finds that it is itself executed from a script, it runs the elaborated path, which does all the mess with copying and appending files. The latter case is no longer necessary with BB 3.9.
Title: Re: THOR's Shell Hacks
Post by: Gulliver on February 06, 2015, 12:52:58 AM
Hi Thomas,
               I have two questions about Shell 45.x:

1) Is it possible for the Shell module to be started with RemLib to avoid reboots?

2) Can this Shell module run in a 68000 Amiga?

Thanks in advance.
Title: Re: THOR's Shell Hacks
Post by: Thomas Richter on February 06, 2015, 01:00:17 AM
Quote from: Gulliver;783145
1) Is it possible for the Shell module to be started with RemLib to avoid reboots?

No, but its possible to use the "resident" command in the startup-sequence to replace the shell module, very much like it was done in AmigaOs 1.3. Then, however, you don't have the boot shell replaced. The shell is both a resident module (more on that later) and an exec module.  

Code: [Select]
resident CLI L:Shell-Seg SYSTEM pure
resident Shell L:Shell-Seg SYSTEM pure
resident BootShell L:Shell-Seg System Pure
Quote from: Gulliver;783145
2) Can this Shell module run in a 68000 Amiga?


Yes. Even though I will get a couple of bad comments from the cycle-counter party, there is nothing specific in the shell that would profit from an 68020 or above, so it has bee compiled for plain 68K.
Title: Re: THOR's Shell Hacks: ViNCed
Post by: Thomas Richter on February 08, 2015, 10:31:10 AM
Ok, now let's have a look at the console the shell runs in. CBMs original console, CON:, was really a pretty poor excuse. It couldn't do much except moving the cursor back and right, and offering a line editor without a history. Then, with Kickstart 1.3, NEWCON was introduced which included at least a history.

At the same time, ViNCEd (back then VNC = VeryNewCon) saw the light of the day, and featured a full screen editor allowing you to move the cursor wherever you want, a history and TAB expansion. For the next issues of this little thread, we'll need this console, namely the version that comes with AmigaOs 3.9. There are earlier releases on the Aminet which are somewhat less powerful.

ViNCed has often been critized for being "not exactly like XYZ", though that was pretty much the point - being different. After all, Amiga is different as well.

So anyhow, here are - as a starter - a couple of instructions how to make it "less different". TAB expansions then go to the console window (instead of a requester) and the cursor up/down navigate in the history (instead of moving the cursor). That's pretty much the settings it came with in Os 3.9, though probably people installed it otherwise.

First, start the Shell and click on "Settings" top right in the window border. SetVNC pops up. Now navigate to the shell page:

(http://www.amiga.org/forums/attachment.php?attachmentid=3990&stc=1&d=1423390833)

For that, click on the "shell" tab (item 1 in the above graphics), then press on the "forward arrow" (item 2 in the above graphics) until you see "Shell page 3 (of 7)" in the display.

Then click on "Requester if expansion is ambiguous" (item 3) which will already give you the matching files on the first "TAB", then press again the forward arrow (item 4). This gives you shell page 4:

(http://www.amiga.org/forums/attachment.php?attachmentid=3991&stc=1&d=1423391020)

On this page, select "List expansions on the console" (item 5, see above), which will disable the requester, but will just print the matching files in the console, and - if you like - "Do not match characters behind cursor" (item 6, see above). This will only match the start of the file upfront the cursor, but not the characters behind it.

Let us now look at the cursor keys. For that, click on the "Keyboard" tab

(http://www.amiga.org/forums/attachment.php?attachmentid=3992&stc=1&d=1423391158)

which is item 1 in the above figure. Then, click onto the box "2", and press the cursor up key on the console. The word "Up" appears in the box. This means that you'll now change the definition of the "Up" key. Finally, click on the forward arrow (item 3) to go to the second keyboard page:

(http://www.amiga.org/forums/attachment.php?attachmentid=3993&stc=1&d=1423391299)

From the list (item 4) select "History up", then click on "Accept" (item 5), then go back to the previous page (item 6) with the backwards arrow.

There, click again into the box, and press on the "Down" key, then go forward again, and select "History down".

Finally, click on "Save" to make the changes permanent, "Use" to test them in all open shell windows, or "To Window" just to test them in the current window.

This gives you a "more standard" shell console. More on all this tomorrow.
Title: Re: THORs Shell Hacks
Post by: Elwood on February 08, 2015, 09:30:24 PM
Quote from: Thomas Richter;782863
the Os 4.x shell is based on the Os 3.9 shell.


This is wrong. H&P never give any source code from 3.5/3.9.
As a result, in OS4 Utilities/IconEdit is the one from 3.1.
Title: Re: THORs Shell Hacks
Post by: kolla on February 09, 2015, 05:25:36 AM
Hah, this means popcorn!
Title: Re: THORs Shell Hacks
Post by: Thomas Richter on February 09, 2015, 07:19:38 AM
Quote from: Elwood;783328
This is wrong. H&P never give any source code from 3.5/3.9.
As a result, in OS4 Utilities/IconEdit is the one from 3.1.

No, H&P didn't, but I did. Remember, the 3.9 shell is not by H&P and was only contributed to H&P under a separate contract. The reason why IconEdit is the one from 3.1 is because H&P did not want to contribute reaction.  Thus, yes, the Os 4.0 shell is pretty much the one from Os 3.9. I don't know how many fixes I made later on where integrated into the 4.x release, but I made sure that Olsen always had access to my code.
Title: Re: THORs Shell Hacks
Post by: olsen on February 09, 2015, 01:26:43 PM
Quote from: Thomas Richter;783373
No, H&P didn't, but I did. Remember, the 3.9 shell is not by H&P and was only contributed to H&P under a separate contract. The reason why IconEdit is the one from 3.1 is because H&P did not want to contribute reaction.  Thus, yes, the Os 4.0 shell is pretty much the one from Os 3.9. I don't know how many fixes I made later on where integrated into the 4.x release, but I made sure that Olsen always had access to my code.
If I remember correctly, we had to start over from scratch for OS4, regarding the components which make up what is commonly lumped together as "the shell".

Those components are 1) the user interface which you use to enter commands, edit command lines, perform copy & paste and which displays the output of commands, and then there is 2) the part which loads and runs these commands. These are Kickstart modules named "con-handler" (1) and "shell" (2).

The major reasons for starting over were in portability (ViNCed was written in 68k assembly language, and would have had to be reimplemented), and the unresolved limitations of the "shell" Kickstart module which, for example, in Kickstart versions 2.04 through 3.1 had a fixed upper limit for the length of an individual command line (the OS4 version has no fixed upper limit).
Title: Re: THOR's Shell Hacks: CONSOLE jobs.
Post by: Thomas Richter on February 09, 2015, 01:54:47 PM
Earlier in this thread we had the question whether * could be substituted with CONSOLE:. The answer is "maybe not, it depends". Actually, if you use * or CONSOLE:, the dos.library does quite the same: It checks for the console process registered for the running process, and forwards the request to there.  

What happens in the console is then up to the implementation of the console, and this "depends". For ViNCEd (see above for installation hints), the usage of CONSOLE: is related to job-control. ViNCEd can run multiple shells in the same window, and can switch between them. There is only one in the foreground, and this shell can print to the console window. All others keep in the background, and once they try to print data to the window, they get suspended. The way how ViNCEd can tell the shells apart is by the "console name" (or rather, how ViNCEd calls it, by the "owner").



So let's try this: Open a ViNCEd shell. For my system, the new shell opens as shell process #7, so it prints for me the following as prompt:

Code: [Select]
7.SCSI:>
This is "shell #7", and the directory is SCSI:, which is the name of my hard disk. Let's start a new shell under a new console owner:

Code: [Select]
7.SCSI:> newshell console:myname &
Now, the "&" at the end tells the shell to detach the next command, i.e. this is similar to "run". Equivalently, we could have used

Code: [Select]
7.SCSI:> run newshell console:myname
The result of this command is a message such as following:

Code: [Select]
[CLI 8]  : newshell suspended. [ViNCEd output]
This means: A new shell had been created, shell #8, and the shell had been suspended because it tried to output something, though currently the active owner is shell #7, the one you are typing in.

You can list the active shells with the status command:
Code: [Select]
7.SCSI:> status
This will print the list of all currently active shells, and the commands they are running. Shell #7 runs the status command (clearly), shell #8 is currently idle (suspended) and runs no command.

Let's switch to shell #8 now:
Code: [Select]
7.SCSI:> fg 8
"fg" is a tiny script in s:. If it is not there, you can also type

Code: [Select]
7.SCSI:> setvnc foreground 8
which is just want the script does. It moves shell #8 in foreground, and shell #7 to background.

The prompt now tells you that you are in shell #8:

Code: [Select]
8.SCSI:>
You can place this shell back into background by "fg 7", or simply "bg", which means, "please put the current shell into background".

Code: [Select]
8.SCSI:> bg
We are now back in shell #7. For creating a new shell, ViNCEd has a short-cut. This is the keyboard combination Control+Z. It only works if the current shell is busy, i.e. you are not able to submit a command. What this keyboard combination does internally is to first pick a unique name, and then call "newshell console:xxxx" with "xxxx" replaced by that name, just to give you the chance to enter a command again.

Now, back to the consoles and the console names. Above, we created a new shell with "newshell console:myname". Clearly, any program that wants to print into "console:newname" or uses this as a path name identifies now itself as part of this new shell. "newname" identifies the "console owner", and the name of this owner is "newname". If you write into "console:newname", and this console owner is currently not in foreground, the command is suspended.

Let's try this: First switch back to shell #7, then print something to the owner of console 8, which is not in foreground:
Code: [Select]
8.SCSI:> fg 7
7.SCSI:> echo "hi" >console:myname
as a result, the echo command is suspended. Because it is suspended, the shell does not return, and you are left without a shell in the window.

What can we do? Well, fetch a free one. This is what Control+Z does, so press the control key, hold it, and press Z, then release both keys. You return to a free shell, which is shell 8.
Code: [Select]
8.SCSI:>
If you place this now into background, finally, the echo command is allowed to print.

Now, if "console:" addresses a named console, what does "console:" stand for? It is the "startup console", i.e. the console that was started along with the window. And what is "*" for? It is the "current console", namely the console that is currently active and in foreground.

Thus, "CONSOLE:" and "*" are not necessarily identical, or rather, they are only identical if the current console is the console that was created along with the window - or the only console.

Thus, it is typically safe to output to "*", as this is in foreground always, and it is not always safe to output to "CONSOLE:" as this console might be in background, and you might get suspended.

Thus, "CONSOLE:" != "*", and the file name behind "CONSOLE:" has a specific meaning, it identifies one out of several consoles that share the same window as output. All the job control and output control is done by ViNCEd, and you can switch between the consoles with the SetVNC program or tiny scripts in S:.

There are, of course, corresponding "DOS Packets" that do the switching, i.e. there is no "magic interface", but rather "job control commands" send to the console the same way output is written to the console.
Title: Re: THORs Shell Hacks
Post by: Thomas Richter on February 09, 2015, 02:06:52 PM
Quote from: olsen;783396
The major reasons for starting over were in portability (ViNCed was written in 68k assembly language, and would have had to be reimplemented), and the unresolved limitations of the "shell" Kickstart module which, for example, in Kickstart versions 2.04 through 3.1 had a fixed upper limit for the length of an individual command line (the OS4 version has no fixed upper limit).

For reference, the 3.9 version does not have an upper limit either. It uses its own string buffering mechanism and self-expanding buffers, i.e. whenever you use a command longer than the current buffer, a new buffer of the necessary size is created. The 3.1 shell had a maximum command length of 512 bytes.

I wonder what happened with all the other shell bugs (e.g. the one related to "skip back" I mentioned above).
Title: Re: THORs Shell Hacks
Post by: ChaosLord on February 09, 2015, 04:28:36 PM
I would like to officially say:

Holy Crap!

I never knew any of this stuff!

I run OS3.9 but mine did not come with any instruction manual.

Maybe there was one on the CD that I never looked at?

The last manual I studied came with my A4000 OS3.0.  If I forget something then that is the manual I autograb off the shelf.  I guess that is a pretty stupid thing to do in 2005+ since there must be a better AmigaDOS manual somewhere...
Title: Re: THORs Shell Hacks
Post by: pVC on February 09, 2015, 04:52:41 PM
Quote from: ChaosLord;783404

I run OS3.9 but mine did not come with any instruction manual.

Maybe there was one on the CD that I never looked at?

The last manual I studied came with my A4000 OS3.0.  If I forget something then that is the manual I autograb off the shelf.  I guess that is a pretty stupid thing to do in 2005+ since there must be a better AmigaDOS manual somewhere...


Doh, yes, there is online manual on the CD. I've copied it to HD to check it always when needed. It's basically structured the same way like the printed 3.0/3.1 manuals and splitted to few separate books. It's really handy to have it available all the time on your machine.
Title: Re: THORs Shell Hacks
Post by: lionstorm on February 09, 2015, 05:46:03 PM
Quote from: ChaosLord;783404
I would like to officially say:

Holy Crap!

I never knew any of this stuff!

I run OS3.9 but mine did not come with any instruction manual.

Maybe there was one on the CD that I never looked at?

The last manual I studied came with my A4000 OS3.0.  If I forget something then that is the manual I autograb off the shelf.  I guess that is a pretty stupid thing to do in 2005+ since there must be a better AmigaDOS manual somewhere...


a nice chap here at a.org scanned them and converted them into amigaguide and pdf : http://www.amigamanuals.co.uk/
Title: Re: THORs Shell Hacks
Post by: Thomas Richter on February 09, 2015, 05:47:51 PM
Quote from: ChaosLord;783404
Maybe there was one on the CD that I never looked at?

There should be a shell.guide (if not there then in Aminet with the shell update) which contains most of the information shown here (except the "Skip lab" episode), and there is the ViNCEd.guide (also in Aminet, part of the ViNCEd distribution) which explains all the above.
Title: Re: THORs Shell Hacks
Post by: Thorham on February 09, 2015, 05:54:42 PM
To Thomas Richter:

Any chance of making the source code of ViNCEd available?
Title: Re: THORs Shell Hacks
Post by: kolla on February 09, 2015, 10:12:57 PM
Personally I find ViNCEd a bit "too much!!" and wish that job control, tab expandation etc could be moved to the shell itself.
Title: Re: THORs Shell Hacks
Post by: ChaosLord on February 09, 2015, 10:38:28 PM
@Thor

Did you program the OS3.9 Shell to use a static memory model?
(That is a memory usage that stays static 99% of the time after it has been started)

I know it will change the amount of ram used if I use a shell command >512 chars but other than that does it go around do malloc();free(); here and there?


The reason I ask is because I do all my compiling and testing from shell.  I test for memory leaks a lot.  Many years ago I found that all the fancy shells that I tested such as KingCON: keep randomly changing the amount of ram that they consume so I can't easily check for memory leaks in my programs.
I think its because of their scrollback buffer constantly expanding.

I like to do
Code: [Select]

avail flush
run myGiantComplicatedProgram blahblah blah moreblah
avail flush

to see if my memory usage is the same before & after.

But that did not work in KingCON: and whatever fancy shells I tested.

I have a vague memory that I may have downgraded my shell back to the old OS3.1 shell many years ago.  I just can't remember exactly.  My history file with all my documented tests was lost in a great hard drive cataclysm :(

I always knew you were a legendary megacoder who does not merely write code but SMASHES code into existence with his mighty magic hammer (+40 vs. bugs) but I had no idea you coded the 3.9 shell.
Title: Re: THOR's Shell Hacks: CONSOLE jobs.
Post by: ChaosLord on February 09, 2015, 10:51:12 PM
Quote from: Thomas Richter;783399

Now, if "console:" addresses a named console, what does "console:" stand for? It is the "startup console", i.e. the console that was started along with the window. And what is "*" for? It is the "current console", namely the console that is currently active and in foreground.

Thus, "CONSOLE:" and "*" are not necessarily identical, or rather, they are only identical if the current console is the console that was created along with the window - or the only console.

Thus, it is typically safe to output to "*", as this is in foreground always, and it is not always safe to output to "CONSOLE:" as this console might be in background, and you might get suspended.

Thus, "CONSOLE:" != "*", and the file name behind "CONSOLE:" has a specific meaning, it identifies one out of several consoles that share the same window as output. All the job control and output control is done by ViNCEd, and you can switch between the consoles with the SetVNC program or tiny scripts in S:.


So for people who are using CLI (that old ancient thing) or OS3.1 Shell it would be accurate to say that for those users * = CONSOLE:  True?

But for people using (OS3.9 shell) or (OS3.1 + ViNCEd)
* <> CONSOLE:

I am just bringing this up because somewhere out there, there is a bug being caused by some script assuming that * = CONSOLE:
Title: Re: THORs Shell Hacks
Post by: F1Lupo on February 10, 2015, 03:03:35 AM
thanks Thomas ! Love this very useful thread:)
Title: Re: THOR's Shell Hacks: CONSOLE jobs.
Post by: kolla on February 10, 2015, 04:21:55 AM
A posting about SPARWED, ie the protection flags, and how H (hold) now works again could be nice, as people seem ignorant about them.
Title: Re: THORs Shell Hacks
Post by: Thomas Richter on February 10, 2015, 06:00:00 AM
Quote from: kolla;783429
Personally I find ViNCEd a bit "too much!!" and wish that job control, tab expandation etc could be moved to the shell itself.

So do I, and ViNCed is even prepared for that. Just that I had never found the time to move this code to the shell. As far as ViNCed is concerned, it would not require a code change. Well, except that I could retire a lot of code with such shell upgrades in effect. More on this later.
Title: Re: THOR's Shell Hacks: CONSOLE jobs.
Post by: Thomas Richter on February 10, 2015, 06:02:44 AM
Quote from: ChaosLord;783438
So for people who are using CLI (that old ancient thing) or OS3.1 Shell it would be accurate to say that for those users * = CONSOLE:  True?
As long as you stick to CON:, yes. As said, it is really up to the console to interpret the "file name", so one cannot generalize this. I believe AUX: also handles * and CONSOLE: alike and ignores the file name, which is the second console (the serial one) AmigaOs has.
Title: Re: THORs Shell Hacks
Post by: Thomas Richter on February 10, 2015, 06:23:15 AM
Quote from: ChaosLord;783434
@Thor

Did you program the OS3.9 Shell to use a static memory model?
No, and for several reasons. You already pointed one out, and that's the dynamic command line length - which can grow arbitrarely large with the 3.9 shell. There are many others. For example, the shell also allocates the stack for the commands it allocates, it allocates memory (indirectly) by loading binaries and opening file handles, and it also creates temporaries in memory for expanding backticks (more on this later) or for the input
Title: Re: THORs Shell Hacks
Post by: Thomas Richter on February 10, 2015, 11:38:39 AM
Ok, let's look today at how the shell executes commands. Sounds boring? You type the command, the shell locates it in the path, and executes it, right?

Well, not quite so. There are a couple of catches. First, there is the "+" sign at the end of the line. It is often misunderstood to continue the current command line at the next line, but then why

Code: [Select]
7.SCSI:> echo hello +
world
does only print "hello" and not "hello world"? On the other hand,

Code: [Select]
7.SCSI:> run +
list
runs the list command, indeed. So what's the difference between "echo" and "run"?

Well, the plus sign also inserts a line feed into the command line, and "echo" simply stops parsing its command arguments there. The rest of the string remains in the standard input of the command, at least if it uses buffered I/O, and waits there to be retrieved after parsing. "Run" parses multiple lines, but all other commands simply stop at the first line feed.

Anyhow, to return to the "+", the following demonstrates this:
Code: [Select]
7.SCSI:> ask "Please enter a number:" numeric to x
The shell will now kindly ask you to enter the number, so do it the pleasure:
Code: [Select]
Please enter a number: 42
Did this number arrive?

Code: [Select]
7.SCSI> echo $x
42
Yes. Fine so. But we can also feed the answer right into "ask" by using this little trick:

Code: [Select]
7.SCSI> ask "What do you get if you multiply six by nine?" numeric to x +
54
At this point, ask doesn't even wait for you to respond, but takes the 54 from the next line and pushes it into its input. Thus, the ask command returns immediately.

There is another magic token like "+", namely "&". This works like "run", actually very much like it, as it executes a command in the background:

Code: [Select]
7.SCSI> list &
7.SCSI>
[CLI 8]  list suspended. [ViNCEd output]
7.SCSI>
Thus, the list command is started, run in background, but as it tries to write something to console, but is not the current console owner (see yesterday's episode), it is suspended. To continue, place it in foreground:

Code: [Select]
7.SCSI> fg 8
If you simply want to run a command in background, redirect its input and output to NIL: Of course, this does not make much sense with list, but anyhow:
Code: [Select]
7.SCSI> list <>NIL: &
Starts list in background, and returns to the shell.

Now, besides commands, we also have scripts. A script is a file whose "s" protection bit is set; these are typically found in S:, for example the "fg" command is one.

If the S-bit is set, the shell does something else. Usually, you would say "execute this as a shell script", but hold on, it's not quite that simple.

It first tries to determine the type of the script. If the first two characters of the file are "/*", i.e. forwards slash, star, then the file name is taken as is, a "rx" is prepended at its start, and the result is executed. Hence, execution is not done by the shell, but by ARexx. These two "magic characters" identify rexx scripts rather than shell scripts.

There are more "magic characters" than /*. These are #! and ;!, both work identical. If a file whose s-bit is set has them as first characters in the first line, the rest of the line denotes the name of a command to execute to run the script, and the rest of the command line would be given to this command. For example, if Amiga had a pyhton interpreter, the initial line "#! C:python" would run the python interpreter. But we can try something similar:

Code: [Select]
7.SCSI:> echo ";! C:ed" >ram:test
7.SCSI:> protect ram:test s add
7.SCSI:> ram:test
This makes the shell believe that "ram:test" is a script whose interpreter is the editor Ed, and by executing it, it gets run with the name of the "script to execute", namely "ram:test", as first argument.

Now, that's the S bit. But there is more. You know of course the feature that the shell implicitly executes a "CD" - change directory - if you just type in the name of a directory or device:

Code: [Select]
7.SCSI:> ram:
7.Ram Disk:>
But what does it do in case the file is not executable and not a directory? Well, even then the shell can do something, though it depends on setting the right variables. So let's help the shell a bit:

Code: [Select]
7.Ram Disk:> set VIEWER SYS:Utilities/Multiview
7.Ram Disk:> Guides:Shell.guide
Now, for that to work, I assume that you have "Shell.guide" in an assign named "Guides:", and that the "e"-bit, for "executable", of this guide is *NOT* set. If it is set, the shell will try to execute the "Shell.guide" as binary, and that of course gives you an "object is not executable".

The way how this is arranged here, however, the shell will determine that the input is a datatype, and will run this datatype through the "viewer", which is "Multiview".  This "VIEWER" variable setting can be made permanent in ENVARC: if you like:
Code: [Select]
7.Ram Disk:> echo "Sys:Utilities/Multiview" >ENVARC:Viewer
and if the above example does not work, remove the "e" bit:
Code: [Select]
7.Ram Disk:> protect guides:Shell.guide e sub
Now, one final "feature" of the shell. Assume that you have the mmu.library in libs:
Code: [Select]
7.SCSI:> list LIBS:mmu.library
mmu.library                       56952 ----rwed 30-May-14 12:01:44
1 file - 113 blocks used
7.SCSI:>
What does the following do?

Code: [Select]
7.SCSI:> LIBS:mmu.library
It prints mmu.library: Unknown command", even though the "command" is right in front of the face of the shell, and it is even an AmigaOs binary, and it even has its "e" bit set, so it should be executable. Why doesn't the shell then try to execute the thing?

The fact is, it actually does. However, libraries and devices are not loaded into the system like this, at least not normally (vnc.library is an exception, hear, hear). So what the "startup code" of the library does is to simply return a "-1" as return code to the shell. And this magic return code, undocumented to my very knowledge, tells the shell "hey, I'm actually not an executable, print an error to the user". Even though it is actually executed, though doesn't do anything useful.
Title: Re: Variable Shell Hacks
Post by: Thomas Richter on February 11, 2015, 12:51:32 PM
How much do you know about shell variables? These work like in any other programming language, REXX for example. You assign them a value like this

Code: [Select]
7.SCSI:> set foo bar

and you retrieve their value a little bit different, namely by putting a dollar sign in front of their name:
Code: [Select]
7.SCSI:> echo $foo
bar

Where do these variables come from? Well, from several places. First, if you just use "set" as above, they come from a shell internal database. This is different for each shell, and hence, such variables are "local" to the shell you use them.

A second type of variables are "global environment variables". They are set with "setenv" instead of "set", but work otherwise the same way:

Code: [Select]
7.SCSI:> setenv hi bar

and you retrieve their value a little bit different, namely by putting a dollar sign in front of their name:

Code: [Select]
7.SCSI:> echo $hi
bar

If two variables of the same name exists, i.e. one environment variable and a local shell variable, the latter is used. Actually, environment variables exist as a file in ENV:, an assign to a directory in the RAM-disk, which is usually copied from ENVARC: during startup. Thus, you can make environment variables permanent by copying their file from ENV: to ENVARC:

Or you can do that in one step:

Code: [Select]
7.SCSI:> setenv save hi bar

the "save" keyword also makes the change permanent and places the variable file in ENVARC: and in ENV:, so it will survive a reboot.

Now, there is a little bit more that can be said about variables? What do we do if we want to write "$foo" on the console, as a literal string, and do not mean the variable? Well, use the BCPL escape-character, which is the asterisk:

Code: [Select]
7.SCSI:> echo "The value of *$foo is $foo"
The value of $foo is bar

Now, here comes the wierd part: Didn't I say that the "*" works only as escape character within quotes? Yes, but... for *$ this is different, it works always:

Code: [Select]
7.SCSI:> echo *$foo
$foo

Why this inconsistency? Well, the original BPCL shell was drilled up over and over again, and apparently, CBM didn't quite know what they were doing, so consistency was apparently not an option, or the inconsistency was not recognized. In the old shell code, parsing and expansion of variables and parsing of strings and quotes was handled in two separate steps, so the code did not even know whether the $ was inside or outside of a double quote.

Speaking of double quotes, did you know that a double quote *within* a string is not understood as a double quote?

Code: [Select]
7.SCSI:> echo thisisa"doublequote
thisisa"doublequote

If there would have been a space in front of the quote, it would have acted as a quote, but this one did not. Every sane programming language would have created an "unterminated string" error for the above line. AmigaDos does not. A quote in the middle of a string is not a quote. Why? Nobody knows...

Back to variables: If you want other characters such as spaces in the variable name, or want to make sure that the variable name is separated from the string, you may enclude the name in curly braces:
Code: [Select]
7.SCSI:> echo ${foo}bar
barbar
without the curly braces, the shell would have tried to expand the variable foobar, which does not exist. If a variable does not exist, it just expands to its name, so let's try this:
Code: [Select]
7.SCSI:> echo $foobar
$foobar
Now, how can we know whether a variable exists? That's what $? is good for:
Code: [Select]
7.SCSI:> echo $?foo
1
7.SCSI:> echo $?foobar
0
7.SCSI:> echo $?{foo}bar
1bar
So $?var expands to 1 if the variable exists, and 0 otherwise, and the last example just showed that curly braces work with $? as well, to separate the variable name "foo" from the rest of the string "bar".

Tomorrow: More on variables. Or actually, things that look like variables, but really aren't. The "alias" command introduces something that works like a variable, but isn't. Again, only CBM knowns why they made this so inconsistent. And then, scripts can take arguments, filled in by "execute". These also look like variables, but strangely enough, they aren't. What CBM was smoking at this time, nobody knows, but it's really another completely different mechanism.
Title: Re: Variable Shell Hacks
Post by: kolla on February 11, 2015, 03:33:25 PM
I recall feeding ssh with password using + many many years ago, something like...

Quote
RequestString password
ssh myhost "remote commando" +
$password

Or on second thought, maybe I ended up using pipe because + did not work with ssh?
Anyhow I felt very dirty, hehe.
Title: Re: Variable Shell Hacks
Post by: kolla on February 11, 2015, 03:38:33 PM
Regarding curly braces - are they default for .bra and .ket now?
Title: Re: Variable Shell Hacks
Post by: Thomas Richter on February 12, 2015, 06:07:06 AM
Quote from: kolla;783636
Regarding curly braces - are they default for .bra and .ket now?

No. ".bra" and ".ket" have nothing to do with curly braces. That's something for today's episode. Confusingly, this is in no relation to shell variables.
Title: Re: Variable Shell Hacks
Post by: pVC on February 12, 2015, 09:26:39 AM
BTW. any idea how can I have parenthesis in destination name/path with Copy command? Using ' characters before parenthesis do work in source, but not in destination. At least in standard shell, haven't tried with Vinced...

For example:

Ram Disk:test> echo "bla" > test(1)
Ram Disk:test> dir
  test(1)
Ram Disk:test> copy test'(1') testfile
Ram Disk:test> dir
  test(1)                          testfile
Ram Disk:test> copy testfile test(2)
Wildcard destination invalid.
Ram Disk:test> copy testfile test'(2')
Wildcard destination invalid.
Ram Disk:test> makedir testdir(1)
Ram Disk:test> dir
     testdir(1) (dir)
  test(1)                          testfile
Ram Disk:test> copy testfile testdir(1)/
Wildcard destination invalid.
Ram Disk:test> copy testfile testdir'(1')
Wildcard destination invalid.
Title: Re: Variable Shell Hacks
Post by: Thomas Richter on February 12, 2015, 12:50:26 PM
Quote from: pVC;783722
BTW. any idea how can I have parenthesis in destination name/path with Copy command?

You can't. )-: It's not even related to the shell, or the console (i.e. ViNCEd). The trouble is the following: The Os pattern matcher pre-parses the pattern to get a more efficient internal representation to do the matching algorithm. That's actually part of the pattern matcher CBM copied/found in arp (the AmigaDos replacement project). Unfortunately, while this pattern matcher correctly implements the apostrophe as escape character (why not the star, it already is the escape character? Who knows...), it unfortunately still tells you "ok, there was a pattern here. A pattern I ignored, though still...". And then copy fails. I'll call this a bug.
Title: Re: THORs Shell Hacks
Post by: ChaosLord on February 14, 2015, 09:22:22 PM
I just noticed that AmigaKit advertises AmigaOS4.1 Final Edition for Classic Amigas with this line:

Quote from: AmigaKit Website

Most powerful console for AmigaOS


Does this mean that OS4.1 uses the THOR-console?
Title: Re: THORs Shell Hacks
Post by: Thomas Richter on February 15, 2015, 10:10:25 AM
Quote from: ChaosLord;784139
Does this mean that OS4.1 uses the THOR-console?

Certainly not. ViNCEd is written in assembly language, and that prevents porting.
Title: Re: THOR's Shell Hacks: Execution and alias
Post by: Thomas Richter on February 15, 2015, 02:37:43 PM
Ok, let's look at today's episode of the shell hacks. We already had variables and how to use them in scripts, but there are more places where elements of the shell are substituted that look like variables - but are not.  First, there is the alias command, a shell-built-in:

Code: [Select]
7.SCSI:> alias ls list


This creates an alias, also "a new name for a command". Now, whenever you type in "ls", the shell substitutes that for you by "list". Where are aliases actually stored? Well, in the same list as local variables, so there have pretty much the same status: They are local to the current shell. Thus, if you want to establish an alias for all shells, you need to put this into S:Shell-Startup where the shell picks it up every time you start one from the workbench.

Aliases also may take arguments. Or rather, a single argument. Without any further preparation, the argument to the alias are simply appended to the contents of the alias, i.e.
Code: [Select]
7.SCSI:> ls df0:
expands to "list df0:" because the alias is extended. If you want to put the arguments of the alias elsewhere, you need to tell the shell where it should put it by using a pair of square brackets. (Why not ${arg} or $* or something? Oh, CBM, consistency, consistency!)

For example:
Code: [Select]
7.SCSI:> alias xd type [] hex
would create a new alias "xd" (probably short for "hex dump") that, when expanded, puts its arguments between "type" and "hex". Hence:
Code: [Select]
7.SCSI:> xd C:Copy
[code]
expands to "type C:Copy hex", and prints out a hex dump of the copy command in C:. Thus, [] is some sort of a variable, but not really, as it only goes for alias and nowhere else.

Can aliases be recursive? Well, partially:
[code]
7.SCSI:> xd ls
expands to (expectedly) "type list hex", because it expands the "xd" alias and the "ls" alias. Unless you are in C: or have a "list" in the current directory, this will of course create an error.

The following, however, does not work:
Code: [Select]
7.SCSI:> xd xd
which will expand to "type xd". The rule is simple: In a single argument line, a given alias can be expanded at most once. This is simply because the shell could otherwise run into an infinite recursion, trying to expand the same alias again and again.

Then, what do you do if you want a literal [ somewhere in an alias? You guess right, put an asterisk upfront. That's the BCPL escape character, and like with variables, it also works here outside of quotes and not only within quotes.

Then, we have a third citizen that has something "almost like but not quite" variables. It is execute, and shell scripts. You can define arguments to shell scripts by defining an "argument template" in the first line of a script:

Code: [Select]
.key filename
type hex

This is a simple script that takes a single argument, denoted "keyname", and then prints the contents of the file in hex, pretty much like "xd" did. Now, what do the angle-brackets do here?

This is, confusingly, not command redirection, but "argument substitution". Again, an inconsistency. "type ${filename} hex" would have been much nicer.

At this point, the shell is completely innocent. The arguments are not even touched or ever seen by the shell. What happens here is all the magic of the "execute" command: It parses the input script itself before forwarding it to the shell, and defines its *own* mechanism for substituting arguments, and does not even use or look at shell variables. In fact, the $ (dollar) sign the regular shell uses to indicate variables has a completely different meaning within execute scripts. It indicates a default value in case an argument is not defined, i.e

Code: [Select]
type hex

uses "C:Copy" as default if "filename" is left blank. Did I say that CBM was highly inconsistent? Did I say that this means that input/output redirection in execute-scripts is hard because angle brackets have another meaning? Probably the author of execute should be ... executed?

Ok, at least one can do something, namely define other characters for such functions:
Code: [Select]
.key filename
.bra {
.ket }
.dollar |
type {filename|C:Copy} hex
which looks much nicer. So you now have "{" and "}" as argument separator, and the vertical bar as default indicator, though that means that you need to spell variables without the curly brackets, otherwise you get nonsense:

Code: [Select]
key filename
.bra {
.ket }
.dollar |
echo "your kistkart version is ${kickstart}"
type {filename|C:Copy} hex
will not do what it should. Yuck!

Dear CBM engineers, this is right away nonsense. The next time you attempt to "design" something, think twice. What about simply putting arguments to scripts into regular variables, and simply use the variables for passing them into commands?

Ok, to be fair, that's not quite how it worked. Execute came first, and it was extended from pure script execution to argument expansion, then came shell variables. Thus, CBM should probably have made variables consistent with execute arguments, which would have meant to loose input/output redirection. No, wait, that's not an alternative either...

One way or another, somebody screwed "execute" and there are only limited means to work around it. Only some parts can be un-screwed by the shell, see the "Skip Lab" episode. Execute is not actually "executing" something. It first runs a very simple substitution algorithm, puts the result of the substitution into T:, and then runs the shell on the parsed result. Instead of manually parsing the arguments, it would have been nicer to leave that to the shell, but then...

While we are at it: How do the ".bra" ".ket" ".key" and ".dollar" commands actually work? Actually, they don't. They don't do anything. If you look at the code for these commands in the shell source, they don't do anything. Instead, they just feed the argument parser of "Execute", so "execute" reads them, and does as indicated, but not the shell. If the shell finds a ".bra", it shrucks with its shoulders and says "ok, good for you, let's ignore this stuff..."
Title: Re: THOR's Shell Hacks: Execution and alias
Post by: ChaosLord on February 15, 2015, 03:10:42 PM
Do you happen to remember what version of AmigaOS first introduced the "Parameter Passing via [] feature of the Alias command" ?

p.s. You have completely melted my brain. :crazy:
Title: Re: THOR's Shell Hacks: Execution and alias
Post by: Thomas Richter on February 15, 2015, 05:13:52 PM
Quote from: ChaosLord;784249
Do you happen to remember what version of AmigaOS first introduced the "Parameter Passing via [] feature of the Alias command" ?
Not exactly. I would guess it must have come in in v37, around the same time the BCPL shell of v34 was replaced by the C shell. V34 does not have it, though it includes the alias list.
Title: Re: THOR's Shell Hacks: Execution and alias
Post by: kolla on February 15, 2015, 05:38:14 PM
If I remember correctly one can also use .bra and .ket many times and write very messy and unreadable scripts :)
Title: THOR's Shell Hacks: Have you sparwed today? Or h is not for hidden.
Post by: Thomas Richter on February 17, 2015, 03:58:46 PM
No, that's not a commercial for a new food fetish. It's on the Amiga protection bits, or at least the protection bits how the (not-so) FastFileSystem defines it and the shell uses it.

Protection bits can be added or removed by the "protect" command, such as
Code: [Select]
7.SCSI:> protect ram:test s add
Sets the "s" protection bit. If you replace "add" by "remove", the bit is removed again.

In total, we have eight of them: hsparwed. The protection bits are partially interpreted by the shell, and partially by the filing system. Now, for the meaning, one by another:

If "d" is set, the file can be deleted, or "unlinked". If it is not set, "delete" on the file fails. This also implies that the file cannot be opened for overwriting, i.e. MODE_NEWFILE for Open fails if the bit is cleared. What it means for directories is pretty much up to the filing system. The current FFS interprets "d" as "the directory cannot be removed", but you can create files in it.

Before we go to "e", first "w": If it is not set, it means that the file cannot be written to. Or rather, almost. You can append to the file, but you cannot overwrite it, at least this is how the FFS currently interprets this, and I would probably call this a bug, because it means that you *can* write to a file, even though w is not set. The RAM: disk has the same issue. Interestingly, this probably changed with the FFS when it was re-implemented in assembler, from the BCPL OFS code.

For directories, "w" should probably imply that you cannot create files within. But it doesn't. Actually, I seem to remember early releases of the FFS did not store any flags for directories.

Regardless of "w", the files or directories can be deleted, so at least a bit of consistency.

"r" means that the file can be opened for reading. At least for files. For directories, one is tempted to say that it means that the directory contents cannot be read, but AmigaOs is not Linux, and consistency isn't its strength anyhow, so "r" doesn't mean anything for directories.

So does "e". Actually, "e" is not interpreted by the filing system at all. It is only interpreted by the shell. If the "e" bit is set, the shell considers a file a "binary executable", and loads it. Or rather, tries to. If that fails, an error is printed. Actually, if "e" and "s" are both set, then "s" takes priority and "e" is ignored.

Actually, the above is not entirely true: If "e" is set in a directory, then of course the shell does not try to execute the directory (how?). Instead, it ignores all other flags and implicitly changes the directory. Or rather, it loads the "CD" command from its built-in command list, and then passes execution to it.

The "rwed" flags are actually stored inverted in the file descriptor - if the bits are clear, they are printed as "set", and vice versa. Thus, if you don't do anything else with the file, the default is "rwed". Actually, not a very good default. "rwd" would have been better as it would have avoided that you try to execute your average text file. CBM at all its glory, once again.

So if what happens if "e" is not set? The shell checks whether the file can be opened by a data type, and if so, checkes whether the variable $VIEWER is set, and then passes control over to the command recorded in this variable. Thus, if $VIEWER is set, for example to "Multiview", you can view pictures, text, AmigaGuides... on the shell by simply typing their names. Ain't that cool?

The next bit is the "a" bit. It actually doesn't do much, except that it is cleared whenever the file is modified. Thus, one can set it by
Code: [Select]
7.SCSI:> protect foo a add
and whenever some program fiddles with "foo", the "a" bit goes away. Sounds useless? No, it isn't. It means that a backup program can skip over all those files whose "a" bit is set, as they haven't been touched since, and while it copies all programs back to another disk, it sets the "a" bit of all the files while it goes. Thus, it only copies what hasn't been modified - quite useful. Actually "SortCopy" (from Aminet) does just that.

The "s" is for "script". That is one of the flags only the shell cares about, and not the filing system. If it is set, the Shell tries to execute the file as a "script", regardless of the "e" bit. Thus, for that, it first checks the first two characters of the file. If they are "/*", the file is passed into AREXX. If they are ";!" or "#!", the script file is passed into the shell command that follows these two magic characters. That's what Linux does all the time, there "#!" is the magic "hash-bang". AmigaOs has it, too.

If neither "/*" nor "#!" nor ";!" is in the first line, then Amiga hands control over to "Execute". Which, as we know now, doesn't do anything special except messing with the file once again by substituting arguments in an completely inconsistent way, and then passing the result back into the shell for execution. Execute doesn't do the dirty job itself. It just creates the mess...

The "s" flag with all following flags are relatively new. The BCPL shell did not have them, and they came in relatively late.

"p" is for "pure". Which actually means not as much as one may think. It was an invention of the "AmigaDos Replacement Project" we've already heard about before: The idea was nice: Why do we have to load commands from the disk everytime again when we need them? So instead, if a command does not change its executable, its data section, and is re-entrant - can be called from itself and by several tasks at once - its code can simply be kept in memory, instead of requiring to load it from disk to memory every time again.

"ARP" had a mechanism for that: The file was checksummed, and if the checksum did not match, it was thrown out the next time. Of course, all that was too much for CBM, so the checksum went away. Instead, it is just a promise. If "p" is set, the command makes the promise "I hereby swear that I will not modify my data and by code. And if I do, I may crash next time". Good job, that's a great stable interface.

Hence, if "p" is set, you can load the command into the dos resident list (more on this later), and the Shell finds it there, and executes it from there rather than loading it from disk. And if "p" is not set, you can set it with "protect" nevertheless and try what happens if you make it resident... Errr, better don't. Only the program author can know that, and hence may set "p". And if it is not set, you better keep your hands off and don't make it resident.

This is how one can make the startup-sequence  a bit faster, namely by moving often-used commands resident:
Code: [Select]
resident C:assign
resident C:copy
So they are not loaded over and over again. And actually, there is an even better way how to do that, without "resident":

Which brings us to the last bit, often misunderstood: "h". Some programs read it as "hidden" and don't list directory entries that have this bit set. That's just wrong, folks. "h" is not for "hidden", "h" is for "hold". What that means is the following:

If a file has the "h", "p" and "e" bits set, so it promises to be pure, and can be executed, then the next time the shell loads the command, it makes it resident all by itself. Thus, the above lines can be avoided in the startup-sequence simply by setting the "h" bit for all those commands you want to keep in memory anyhow, and you planned to make resident, except that you don't need a resident command anymore.

Unfortunately, due to size constraints, this nice feature was "thrown out" of the V40 shell in last minute, so the Os 3.1 shell does not support it. The 3.9 version does, as there is no ROM size constraint, and "h" really stands for "hold".
Title: How would you like your console today? Rare, English, Medium or Well Done?
Post by: Thomas Richter on February 20, 2015, 01:27:14 PM
Today, I'd like to write about the console again, or what is known as CON:, RAW: or VNC:. Actually, you may know the two console handlers AmigaOs has to offer: CON: and RAW:.

For CON:, every line is buffered, the device sits and waits and gives the user the chance to edit the full line, until he presses return, and at that point, the entire line is transferred to the program that waits for input; usually the shell, you name it. This is also called the "cooked" mode, as user input is "well prepared until it's served", namely to the receiving program. That's a name that comes from Unix, and Amiga has picked that up.

Then, we have RAW: Unlike CON:, it does not provide any type of line buffering. Every single keystroke is transferred to the receiving program, and the keystroke is not even printed on the screen. Instead, the receiving program has to do that. Since the AmigaShell does not, RAW: is not useful for the shell. Some third party shells provided better editing facilities, and then did so by implementing all history and editing themselves, building upon RAW: rather than CON:, because CON:s editing services are (or were) rather limited. At least until ViNCEd, which offers a very rich editor and does a lot more than CON: for that matter.

What is probably less known is that CON: and RAW: are two "flavours" of the very same console handler. One can open a console handler in "raw" mode, and switch it to "cooked" mode by a specific command, and also go back. This "almost" works, for CON: at least. In a RAW window, a program can, for example, ask the console to send information whenever the user resizes the window. In cooked mode, this does not work. In raw mode, a program can ask the console to send a string when the user clicks somewhere. In cooked mode, this does not work.

Or at least, did not work. ViNCed, of course, is no difference, and can be both "raw" or "cooked" - or as it is called here "rare" or "well done", as I you don't "cook" steak. In ViNCEd, this stuff works right, and you receive such messages, no matter whether the console is "raw" or "cooked" - ehem - "rare" or "well done". If a program asks for information on window resizes, it gets them.

Of course, ViNCEd's "cooked", ehem, "well done" mode is really pretty rich: It not only offers line buffering, but full screen editing. It also does TAB expansion, and job handling. It keeps a history of commands. That's probably all stuff that should better go into the Shell, as it is the matter of the shell (and not the console) what is going on at the command line. It didn't, because the AmigaShell evolved slower than the console.

As I did not like this architecture either, and wanted to open the possibility to relocate tab expansion, history, and job control into the shell, ViNCEd has four modes instead of two: "rare", "english", "medium" and "well done".

"Medium" is slightly less cooked than "well done": The console still does the full sceen editing handling, but leaves TAB expansion, job handling and history to the shell. Thus, what happens here is that - instead of expanding file names with TAB - the console picks up the file name to be expanded from the console, places the file name into a control sequence, and forwards the request to the Shell. Same goes for job control and history. Thus, it is the shell that has to receive such control sequences and act accordingly.

Of course, the AmigaShell is currently unable to do that, so the "medium mode" remains currently unused.

Then, we have the "English" mode, which is slightly more cooked than "raw", ehem, "rare". This mode works very much like the raw mode, except that the "shell features" such as TAB expansion, history and job control get special control sequences, allowing the user to relocate such functions to other keys on the keyboard, and the shell (again) receives such keyboard sequences instead of the "pure raw" sequences. Thus, if the user presses TAB, a "raw" console would send the ASCII character for TAB (08). In the "english" mode, the console would rather send a control sequence indicating "Hey, the user wants to do TAB expansion", and if the user redefines the keyboard, the same function would be bound to - for example - F1, and TAB would continue to send the ASCII character for the tabulator.

Thus, "english mode" distinguishes between "keyboard functions" that can be re-bound to other keys, and "raw keyboard codes" that remain bound to the native keys, unless overridden by a function. Keyboard redefinition thus remains, in English mode, to the console, execution of such sequences to the shell.

Of course, this mode is also completely unused as of today. It is just there.

Now finally, how does the console actually distinguish all these modes? Essentially, it is in the mount list, i.e. the part in DEVS:DOSDrivers that defines how a handler is loaded. For the old CON: console, this file is in ROM (or rather, its binary equivalent). For ViNCEd, it is in DEVS:DosDrivers. For the "well done mode", this file looks as follows:

Code: [Select]
Handler     = LIBS:vnc.library
Priority    = 4
StackSize   = 4096
GlobVec     = -2

Interestingly, the handler does not sit in L:, but rather in LIBS: because it is *also* a library (a neat trick, there is only one file for both functions, which is entirely possible).

For a "raw" console (typically called VNR: instead of VNC: or NEWCON:), it reads as follows:
Code: [Select]
Handler     = LIBS:vnc.library
Priority    = 4
StackSize   = 4096
GlobVec     = -2
Startup     = 1

The difference is in the "startup" line. Startup = 0 (or no startup) is "well done", startup = 1 is  "raw".

Startups "2" and "3" are the two new hidden modes.

Actually, the same trick, namely to use startup to switch between various "flavours" is also used in different edge of the operating system. SER:, PAR: and PRT: are all implemented by the same handler, namely the Port-Handler. It sits in L: The only difference between the three flavours of it are the values of "Startup". Of course, you usually don't see that because there is no entry in the mount-list. Instead, there is a binary equivalent of this mount entry built into the ROM.

There is small program, though, which displays these mount settings. It is called "Devices" and can be found in Aminet. It shows you what the entries look like, even without an explicit mount list to look at.
Title: Re: THOR's Skip Lab
Post by: Boot_WB on February 20, 2015, 02:20:04 PM
Quote from: Thomas Richter;783069
Today, I'll try to cover a bit a dark edge of the Amiga Shell, and these are the two commands "SKIP" and "LAB", and why they sometimes didn't work before 3.9. Of the two, LAB is actually not a command at all - or rather,  it is, but it doesn't do anything, while still being necessary.

Flow control - that's a key feature of every programming language. That is, to control where program flow goes. "if-then", "for-next", "do-while", "repeat-until", this kind of stuff. Unfortunately, the only thing AmigaOs has is "if-then", and "goto", even though the latter is pronounced "skip". It's the same ugly command that has been banned from many more reasonable languages for good reason - it creates spagetti code.

Anyhow, it's all we got, so here is how it works: If you have a "lab foo" somewhere in a shell script, then a "skip foo" continues control to this very position in the script. Simple enough? Not quite. If the label is *above* the skip command, i.e. on a line "higher up in the code", you need to say "skip BACK foo".

Simple enough? Not quite. If there is an "execte" somewhere between the label, or you are calling a shell script by some other means - probably because you have a file whose 's' bit is set, it doesn't work at all. Or at least, it did not *used* to work before Os 3.9 latest BoingBag.

Why now does the freaking shell care about execute or not? That's again one of the sad stories of "bad design decision". Remember, commands have an input stream, and an output stream, you can redirect with "<" or ">". In  a sense, the shell has the same: It has an interactive input, which is the console the user types on and the shell connects the input of commands it executes to. This is the console. This stream is used for the "ASK" command, for example, to read user input. But the shell has *also* a command input, and this is where the commands for the shell come from. That's usually also the console, as the shell executes the commands as you enter them.

Now, however, if you execute a script (explicitly, or implicitly by a set s-bit), the shell does the following: It connects its "command input" to the file of the script. Sounds like the easiest thing to do: Magically, the input comes now from the script file. Since the interactive input is still connected to the console, the "ASK" command and all other commands still receive that as standard input, and hence read their input from the console as they should.

That's all pretty lean and neat unless... you want to execute a script from a script. Ooops, what do we do now, the CBM engineers said? They came up with the bloddiest ugly hack you can think of: Take the script to be executed, copy that to the RAM-disk (actually, T:), and append the rest of the script that is currently executed (and hence, the script that calls the other script) and append that at the bottom, then execute the new script.

That's just so wrong. I've written some hacky scripts, but even I wouldn't consider that a 'solution'.

Quote
See where I'm getting? Since the top of the current script is lost, and hence any "LAB"  command is lost, a "skip back" to a label that is below any execute command cannot see the "LAB" anymore, and hence, cannot skip back.

Luckely, the 3.9 shell is smarter, and does what every proper programming language does with recursion: It keeps a stack. Thus, yes, you can "skip back". In case you care, the stack is kept in some shell variables (you can see them with the "set" command, to be discussed in a later edition).

What the 3.9 shell does not is to "get away" with skip, and introduce *useful* flow control, as in "for" or "while". It would not be overly hard to do, but simply was not done. A bit of tradition still had to remain, I'd say.

Excellent summary Thomas, thankyou.

WaitX (Aminet) is also damned useful for creating cleaner scripts if anyone needs a bit more flexibility than 'wait' provides (often used with skip...lab).
Title: Re: THORs Shell Hacks
Post by: TuKo on April 20, 2016, 06:09:58 PM
Dear Thomas,

I'm from the Apollo team and I'm following with attention your work on kickstarts modules.

As we are thinking to build a better ROM than we have today to give more functionnalities and bugfixes to users, I'd like to ask if you would kindly agree if we use your patches in our ROM ?

I mainly think :
* Shell 45.29
* layers.library 45.27

Thanks in advance
Title: Re: THORs Shell Hacks
Post by: Thomas Richter on April 20, 2016, 07:35:26 PM
Quote from: TuKo;807376
Dear Thomas,

I'm from the Apollo team and I'm following with attention your work on kickstarts modules.

As we are thinking to build a better ROM than we have today to give more functionnalities and bugfixes to users, I'd like to ask if you would kindly agree if we use your patches in our ROM ?

I mainly think :
* Shell 45.29
* layers.library 45.27

Thanks in advance

I have personally absolutely no problem with that, but you should also know that I'm not the owner nor full author of these modules. So I'm unfortunately not in the position of giving you (or denying you - which I wouldn't) access to these modules. So if in the end, hell breaks loose for whatever reason, my "allowance" helps nothing at all, and would be premature and even misleading.  

The shell-segment was done as part of Os 3.9, from where I currently derive my allowance to provide patches (but not full versions) of the shell. To my very knowledge, the IP of the Shell went back to Amiga, and from there to Hyperion, which also bought (to my knowledge) the rights on the 3.9 parts.  

Layers V45 was developed originally as part of Os 4.0, and hence was or is probably Hyperion's IP to begin with. At least the Os 4.xx code has a lot in common with the V45 layers as we have it today.

So in the end, I believe you need to talk to Hyperion to be really in a legally sound position, and you would *probably* need to make an arrangement with them to get a license to distribute the kickstart.  What I can do - if this is of any help - is to arrange a contact with them (and no, unlike what many believe, they don't eat children. Or poor developers for that matter.).  

I believe it's probably best if you'd approach me in private over my regular mail address. It hasn't changed over the years. You'll find it in the shell guide. Or elsewhere.  

  Please don't take this as a "no";. It really isn't - it's rather "I'll try my best to make it a yes".  

  I've a lot of respect what your team does, it's the best thing that happens to the platform in years. I'm simply not in the position of giving you something I do not own, and there are more people in the boat than you would probably imagine. Let's simply talk and I believe we'll find a way.
Title: Re: THORs Shell Hacks
Post by: TuKo on April 30, 2016, 07:09:08 AM
Dear Thomas,

I'd like to thank you for taking time to answer my request. I fully understand your point.

We are actually focusing on testing and being ready for the V500 which is coming soon so the topic for an upgraded kickstart is left a bit behind atm.

Cheers
Title: Re: THORs Shell Hacks
Post by: Thomas Richter on January 12, 2017, 08:59:10 PM
Probably in time to expand this tutorial a little bit by more "trivia". Did you know that you cannot redirect the output of "Execute" to a file?  

Thus, if you write a small script as follows
Code: [Select]
.KEY DIR

List <DIR>
save that as "T:myList" and then "Execute T:List", then - probably surprisingly - a

Code: [Select]
Execute T:myList SYS: >results

does not redirect the output of the script to "result". Instead, the directory listing just appears on the terminal. Weird?

Well, maybe, unless you know why that is. The point is that it isn't actually "Execute" that executes the script. Not at all. Execute just parses the script (incorrectly, actually, until recently), creates a temporary file in T:, and then leaves it to the shell to do the execution. So all the "gory work" is left to the shell. Execute is not doing very much at all.

By the time "Execute" returns, the execution of the script has not even started, but output redirection is un-done by the shell before the shell even attempts to read the temporary file in T:

The same goes of course for input redirection, or console redirection (the thing with "*>", see above).

Oh yes, at least until recently. I fixed that. Make "C:Execute" resident (ideally in the startup-sequence) , then open up a new shell, and then (and only then) input and output redirection works, even for Execute. That's more a workaround than a fix, given the above implementation, but at least one annoyance less.

By the way: The above script ("T:myList") contains at least potential problem. Do you see it? Can you fix it?

Probably more on this next time...
Title: Re: THORs Shell Hacks
Post by: kolla on January 13, 2017, 12:12:11 PM
Quote from: Thomas Richter;819623
By the way: The above script ("T:myList") contains at least potential problem. Do you see it? Can you fix it?

Not sure, please do tell :)

The way Execute parses the scripts can be used for funny exercises , such as changing .BRA and .KET conditionally throughout the script, the parsed script may very well look very different each time the script is executed.

One thing I have been missing, is a way to do "set echo on" in startup-sequence, but with output to a file in RAM: - for example if the echo variable could be a filename "set echo ram:bootlog"?
Title: Re: THORs Shell Hacks
Post by: Kernel on January 13, 2017, 04:50:58 PM
Quote from: Thomas Richter;819623
Probably in time to expand this tutorial a little bit by more "trivia". Did you know that you cannot redirect the output of "Execute" to a file?  

Thus, if you write a small script as follows
Code: [Select]

.KEY DIR

List <DIR>

save that as "T:myList" and then "Execute T:List", then - probably surprisingly - a

Code: [Select]

Execute T:myList SYS: >results


By the way: The above script ("T:myList") contains at least potential problem. Do you see it? Can you fix it?

Probably more on this next time...


DIR is a command...?
Title: Re: THORs Shell Hacks
Post by: amoskodare on January 13, 2017, 06:40:09 PM
In that script DIR is the given name to a variable, which holds the parameter from Shell, which is passed on to the List command (if I'm not totally incorrect).
Title: Re: THORs Shell Hacks
Post by: Thomas Richter on January 13, 2017, 07:29:10 PM
Quote from: kolla;819645
Not sure, please do tell :)
The problem appears as soon as you pass the script a parameter that contains spaces. So for example, try the following:

Code: [Select]
execute t:myList &quot;Ram Disk:&quot;
The problem is the following: Execute uses for parsing the arguments "ReadArgs()", similar to all other shell commands. Actually, it does so since Kickstart 1.2, or even since Tripos. (Note: ReadArgs(), even though it started to appear in Kickstart 2.0, is part of the dos "GlobVec" since the Tripos age, but was not exposed to AmigaOs before release 2.0. Anyhow...)

So, let's look at what ReadArgs does. It takes the entire argument, thus, "Ram Disk:" and places that in the argument list of Execute. Then, Execute puts that into "DIR". Hence, which value does "DIR" then have:
Code: [Select]
Ram Disk:
Obviously. Now, if that is inserted into the script, substituted for , we get the following temporary file:
Code: [Select]
List Ram Disk:
and this is what is passed to the shell. Do so see the problem?

Quotes are missing! Hence, what "List" sees are two arguments: "Ram" and "Disk:". And of course, "list" will error, because there is (likely) no file named "Ram" in the current directory, and no device or volume named "Disk:".

Bummer!

So, what can we do? Probably the following:
Code: [Select]
.key DIR

list "
"
So, just deliver the quotes, and everything is fine. "Ram Disk:" is again quoted, and a single argument, and if the file name is something without quotes, such as "SYS:", adding the quotes will not harm.

Really? Actually, not. And this is where the Execute V37.11, which is delivered with Os 3.1 stopped working and broke.

To see the problem, consider the following: You had enabled using the asterisk ("*") as wild card, so that
Code: [Select]
list *
will list all files.

Now try the follthe following:
Code: [Select]
execute t:myList *
and this won't work... "Unmatched quote", at least with the original "Execute", and even the one with 3.9. What is going on again?

Well, the original stupid execute replaces again the parameters with the arguments, and then left the following to interpret to the shell:
Code: [Select]
List "*"
The quotes are from your original quotes. If you do not see the problem with that remember that the asterisk is the BCPL escape-parameter, which only operates in quoted parameters - which we find here - and the combination *" means "a literal quote". And hence, the initial quote is not matched, and the shell errors.

Bummer! Similar problems can be constructed with file names like
Code: [Select]
Thisis"atest
or
Code: [Select]
AnotherStupidExample"
all of which are, surprisingly, completely valid and syntactical correct. A quote within a string is a literal quote, and does not quote anything (stupid, stupid shell syntax!)

If you quote that again, the literal quote within the strip becomes a functional quote, and messes the string up completely.....

Oh well - great job! Now, that's exactly what the new execute (V45.14) is good for. It really tries to understand what the string is, and escapes arguments correctly within quotes. With 45.14, the above example becomes
Code: [Select]
list "**"
instead. Execute notices that it has to substitute an argument within quotes, and hence has to escape the asterisk with another asterisk, and this - again - does the right thing. ReadArgs() will parse off the argument, will notice that it is quoted, will remove the quotes and re-interpret the "escape character" *, and replaces thus "**" by *. Which is what we want.

Similarly, the considerably wierd file name Thisis"atest becomes then
Code: [Select]
list "Thisis*"atest"
where *" is the escape sequence for the literal double quote.

Thus, Execute V45.14 learned how to parse the shell syntax, and place arguments back into quotes with proper escape characters in place. Which is also the reason why it is almost twice the size of the V37.11 version.

Hence, remember to quote execute parameters, and use a working execute.

(If this all causing a headache, you are not alone. I got a headache, too...)

Quote from: kolla;819645
The way Execute parses the scripts can be used for funny exercises , such as changing .BRA and .KET conditionally throughout the script, the parsed script may very well look very different each time the script is executed.
Indeed. I know the Bantam book and the manual actual gives examples like this, but whether that's all intended is the next question. I would really keep the "dot commands" to the start of the script and I would believe that this more works by accident than by design as it may actually backfire (as the quotes) to the script.

Now that the shell learned to understand "input file inclusion" with <

Quote from: kolla;819645
One thing I have been missing, is a way to do "set echo on" in startup-sequence, but with output to a file in RAM: - for example if the echo variable could be a filename "set echo ram:bootlog"?
Yes, right. One of the many old problems the shell has. "Echo" is pretty much legacy and probably ready for another update round. Though it's not exactly getting simpler....

I'd probably tell next time how the execute "Key" looks like. In the mean time, try to get a ".KEY" such as ".KEY BIRTHDAY/N" working. That, again, is another problem left over from Tripos. As in many other places.

So long,
Thomas
Title: Re: THORs Shell Hacks
Post by: kolla on January 14, 2017, 05:53:19 AM
Ah yes, paths with spaces, of course. Btw - I read the guide for your latest shell update, and the part about $STACK: it says it expects decimal as value, but surely you meant to write integer?
Title: Re: THORs Shell Hacks
Post by: Thomas Richter on January 14, 2017, 09:18:06 AM
Quote from: kolla;819692
I read the guide for your latest shell update, and the part about $STACK: it says it expects decimal as value, but surely you meant to write integer?

Hmmm? It expects certainly an integer, but encoded in ASCII to the base of ten, i.e. a decimal number.
Title: Re: THORs Shell Hacks
Post by: kolla on January 15, 2017, 01:22:14 PM
Quote from: Thomas Richter;819696
Hmmm? It expects certainly an integer, but encoded in ASCII to the base of ten, i.e. a decimal number.

Yeah, well - but decimal numbers may contain fractions, for example 11.37 :)

No big deal though, just confused me a little.
Title: Re: THORs Shell Hacks
Post by: kolla on January 15, 2017, 01:39:46 PM
No really related to the shell itself, but anyhow... Something I have pondered on what/what defines LFORMAT entities for c:List, for example? And why isn't there one that is like %P, only without trailing / ? :) Nor is there an option to list a drawer itself instead of the content of a drawer - right? I am missing the equivalent of *ix "ls -d".

Also it would be nice to override locales, just today I was taken off by how I have to use norwegian like
Code: [Select]
list all files since &quot;i dag&quot; ("i dag" meaning "today") - I don't mind that I _can_ use norwegian, but it would for sure be nice, for scripts etc, that english worked as well. Same for input to C:Date.
Title: Re: THORs Shell Hacks
Post by: Thomas Richter on January 15, 2017, 02:17:18 PM
Quote from: kolla;819768
Something I have pondered on what/what defines LFORMAT entities for c:List, for example? And why isn't there one that is like %P, only without trailing / ? :) Nor is there an option to list a drawer itself instead of the content of a drawer - right? I am missing the equivalent of *ix "ls -d".
This is a particular shortcoming, and probably due "list" just using the arp pattern matcher (now part of dos.library). If you think about it, "LFORMAT" is neither bullet proof.

So, for example, what happens if your directory contains a file whose file name includes a quote or an asterisk? While I have not checked, I would *bet* that "list LFORMAT" forgets to escape such characters properly, i.e. a file name including a quote with an LFORMAT that includes its argument in quotes as well should replace " by *" and * by **.

It is really astonishing how little thought the creators have put into the syntax of the shell.

I don't think "list" can be as easily replaced as "execute" due to its many options.

Quote from: kolla;819768
Also it would be nice to override locales, just today I was taken off by how I have to use norwegian like
Code: [Select]
list all files since &quot;i dag&quot; ("i dag" meaning "today") - I don't mind that I _can_ use norwegian, but it would for sure be nice, for scripts etc, that english worked as well.

It would not be only "nice", it is a necessity to allow portable scripts that work independently of the locale.

As you note correctly, there are lots of problems with the shell, and it will take a while to clean up all the mess. It seems I've just scratched the surface.
Title: Re: THORs Shell Hacks
Post by: Thomas Richter on February 01, 2017, 07:59:27 PM
Back to "execute" - as announced a while ago. As you probably know, shell scripts may take arguments, and quite like "real" shell commands, shell scripts also use a "template" that defines the type of the arguments, whether they are required or optional, whether they are switches, toggles and similar things. That's what the ".key" pseudo-command on the first line of a script does:

Code: [Select]
.KEY FROM/A/M,TO/A,CLONE/S

copy <FROM> <TO> <CLONE>
would be a simple wrapper to the "copy" command, where "FROM" is a mandatory argument, and multiple of them can be specified. "/A" means "mandatory", and "/M" means "one or more of them". "TO" is also a required argument, and "CLONE" is a switch.

Identical to binary commands, the letter behind the slash defines the type and requirements of the argument, and thus
Code: [Select]
.KEY YEAR/N

echo "We are in year "
would take a numeric argument, and refuse anything that is not a number.

That is how it is documented, and how arguments work. Execute uses the dos.library "ReadArgs()" function to parse off arguments, so all the nice "/" types come for free.

Unfortunately, however, none of the above examples work if you try them with the original v37 "Execute" command, and that is the one that came with workbench 3.1. Now, how come? Didn't I just say that "Execute" uses the dos.library "ReadArgs()", and that for this reason argument parsing should just work, because "ReadArgs()" surely works? Why doesn't it work for "Execute" then?

Well, this requires a bit of background. What ReadArgs() does is that it fills an array of string pointers, one for each argument. Or at least, this is what "rdargs()", the BCPL counterpart in Tripos did back then in the Kickstart 1.3 times. So the BCPL "Execute" from 1.3 did just work.

The problem is that kickstart 2.0 added a couple of new argument types. So for example, "/N" for "numeric argument" is new, and "/M" for "multiple arguments" is new. It did not exist in 1.3, and these two types were added by "arp", the amiga replacement project, and became later part of AmigaDos.

What "arp" also did is that it changed a bit the convention of how "ReadArgs()" works. So for example, for a "/N" argument, it is of course much more convenient to return the number directly instead of a string, so "ReadArgs()" fills in a pointer to a LONG instead of a pointer to a character array. This is certainly much more convenient than having to parse the number *again* in the program - ReadArgs() can do that already.

Similarly, for "/M" ReadArgs() does not leave a string pointer in its return vector, but rather, returns a pointer to a string array, i.e. a pointer to a pointer of characters. It has to, since "/M" may take multiple arguments on the console, so one string is not enough.

Unfortunately, nobody told "Execute" on the changed syntax, and nobody dared to touch the execute sources either. Thus, if the ".KEY" contains a "/N" argument, the v37 execute (the official 3.1 one, note that there is no v40 execute) expects a string pointer in the arguments, but just finds a pointer to a number there. And does strange and wonderful things, of which an "Enforcer hit" is probably the most harmless. Outch!

Hence, someone (the software distillery, namely) changed the interface of an Os function behind the "back of Execute", and thus broke a command. Or rather, they forgot to update Execute.

Unfortunately, this change means that now execute also needs to be aware of the type of arguments it has to parse off, and has to convert the numbers or multiple strings back to one single string, namely those that it requires for substitution of the parameters of the script, i.e. the things enclosed in angle brackets.

Hence, for V45 execute (and also V44), a couple of things unfortunately happen twice: First, ReadArgs() converts strings to numbers for /N arguments (because it always does), and "Execute" has to convert them back (for parameter substitution) such that the command in the script has to convert them back from string to number again.

There are more types like this: "/S" just leaves a boolean flag instead of a string, so "Execute" has to re-insert the name of the switch instead of the boolean value. The same goes for "/T", which is actually not really a toggle, but takes two values, "ON" or "OFF", or alternatively "YES" or "NO". The documentation is not quite appropriate.

"/M" returns a pointer to a string array, so "Execute" has to merge all the strings together.

And last but not least, do not forget that Execute also has to take care of proper quoting arguments with blanks in it - and escaping of the asterisk and the quote in the strings itself. We had this before.

So, it's actually a bit more complicated than it seems.
Title: Re: THORs Shell Hacks
Post by: kolla on February 02, 2017, 12:53:36 PM
Can you say something about when Execute is used, and hence what shell features one can use in the different cases? For example, it is not used to execute S:Startup-Sequence, but it is, by default used for S:User-Startup. How about S:Shell-Startup? :)
Title: Re: THORs Shell Hacks
Post by: Thomas Richter on February 02, 2017, 03:37:16 PM
Quote from: kolla;821345
Can you say something about when Execute is used, and hence what shell features one can use in the different cases? For example, it is not used to execute S:Startup-Sequence, but it is, by default used for S:User-Startup. How about S:Shell-Startup? :)

When a program calls "SystemTagList()" or "Execute()" (the dos.library call), then the "Execute command" will *not* be used (ok, unless of course it is a command in the script being executed - clearly). The commands will just go to the shell directly as "command input". This happens for the startup-sequence, and also for s:shell-startup and s:cli-startup. In the first case, it is the dos.library that (implicitly, not directly) initiates the bootstrap shell. In the two other cases, it is the Sys:System/CLI binary that simply calls "Execute()" (the dos.library function call, not the command)".

The "Execute" command *is* used if you either call it explicitly (clearly, as for the "User-Startup") or if the S-bit of a script is set and the shell identifies it as shell script.

Even in the latter case, one cannot really say that "Execute executes the script". Not at all. At best, it performs argument substitution, and then just "bends a couple of pointers" of the current CLI to leave all the dirty work to the shell.

"Execute" is really a confusing name for this binary. It's more the "CLI command input stream redirection" binary, but that would have been too long a name, I suppose. A better command name would have been "<". If you understand this joke, you got the shell syntax and the idea how execute works.
Title: Re: THORs Shell Hacks
Post by: patrik on March 25, 2017, 09:16:12 PM
Quote from: ChaosLord;783434
@Thor
The reason I ask is because I do all my compiling and testing from shell.  I test for memory leaks a lot.  Many years ago I found that all the fancy shells that I tested such as KingCON: keep randomly changing the amount of ram that they consume so I can't easily check for memory leaks in my programs.
I think its because of their scrollback buffer constantly expanding.

I like to do
Code: [Select]

avail flush
run myGiantComplicatedProgram blahblah blah moreblah
avail flush

to see if my memory usage is the same before & after.

But that did not work in KingCON: and whatever fancy shells I tested.


Atleast with KingCON this works fine if you just disable the "Review"/scrollback in the menu. Otherwise you will loose memory for every additional line.
Title: Re: THORs Shell Hacks
Post by: kolla on December 04, 2017, 10:19:43 AM
Digging up this old thread... :)

@ThoR

Instead of writing, let me just illustrate my question...
(http://www.amiga.org/forums/picture.php?albumid=204&pictureid=1433)
(and this doesn't even show all my desperate attempts at escaping)

I hope it is obvious what I try to accomplish? :)

My goal was to have the content of a file displayed as the body of RequestChoice.
Title: Re: THORs Shell Hacks
Post by: Thomas Richter on December 04, 2017, 01:18:10 PM
The problem is that you escape improperly and at the wrong place. So let's start...

Code: [Select]
set blarb foo*nbar
sets the variable "blarb". However, the * is not in quotes, so it is a literal. Nothing will change that any more. So if you now print the contents of the variable or use it

Code: [Select]
echo ${blarb}
then this will print an asterisk, simply because there was an asterisk in the variable in the beginning.

If you try this:
Code: [Select]
echo "${blarb}"
then the shell will notice "Oops, he's putting the variable into quotes, so I need to escape its contents". And so the shell does. It detects that there is an asterisk in the variable contents, so it needs to escape this asterisk, and replaces it by two asterisks before passing it to echo, which will run into "ReadItem()" and again, replaces the two asterisks back to one.

If you want a line feed in a variable, you need to put it there in first place:

Code: [Select]
set blarb "foo*nbar"

*Now* the asterisks is in quotes, and means "escape character". And you can use the variable as such:

Code: [Select]
echo ${blarb}

or as such

Code: [Select]
echo "${blarb}"

and the line feed will stay a line feed. (-:
Title: Re: THORs Shell Hacks
Post by: kolla on December 04, 2017, 02:28:10 PM
Hm, so how do I make a variable that can subsitute "foo foo*nbar bar" in the below example, so that the requester pops up with two lines in body, like it does in the below example? I really tried using all your suggestions, and except for the last one, where it only expands to "foo foo" (the first line before line feed), all I ever get is literally "foo foo*nbar bar" (without quotes) in the body :)  

Code: [Select]
C:requestchoice title &quot;test title&quot; body &quot;foo foo*nbar bar&quot; gadgets &quot;OK&quot; &quot;Cancel&quot;

And worse yet - how to get the content of a file there, with linefeeds represented by "*n", using backticking? :)

Something that works, but isn't quite what I picture a solution...
Code: [Select]
set blarb foo foo
set blurb bar bar
requestchoice "testing" "${blarb}*n${blurb}" "OK"

so... with one variable per line, it works. But that is very cumbersome :)
Title: Re: THORs Shell Hacks
Post by: Thomas Richter on December 04, 2017, 09:54:48 PM
Frankly, I'm not sure whether that is a bug or a feature. The problem here is that a variable with a line feed in the middle is cut at the line feed whenever you try to insert it.

Now, at first sight, this looks like a bug, and it is part of the GetVar() interface of the dos.library, not the shell (directly), which truncates variable contents at line feeds - unless you ask for "binary variables".

Well, has this been done? One can only guess, but consider the following case:

1> echo "foo" >env:bar

This sets the environment variable $bar to "foo", right?

Wrong. It sets it to "foo*N", which is not equal to "foo".

So in this case, it *seems* sensible to cut the variable contents at the *N.

In other cases, it is not. With the current shell interface, it will get the variable "to what you would expect to be right", but in other cases, truncation of the variable contents at a line feed is not what you want.

I do not yet have a clear answer to this problem and need to think about it.
Title: Re: THORs Shell Hacks
Post by: kolla on December 05, 2017, 02:00:32 AM
Thanks for all the thoughts and work you put into the shell updates!

And woop - found the solution - quotes inside the file :)

A file with content (including quotes):
Code: [Select]
&quot;foo foo*nbar bar&quot;and
Code: [Select]
echo `file`
vs a file with content (no quotes):
Code: [Select]
foo*nbarand (with or without quotes, makes no difference it seems)
Code: [Select]
echo "`type file`"
The first does work, the latter doesn't.

So, success - now I know how to alter files to show up as multiple lines with RequestChoice (still center aligned... hm) :)

(http://www.amiga.org/forums/picture.php?albumid=204&pictureid=1435)

(I really do an effort to just use ASL and not ReqTools etc)
Title: Re: THORs Shell Hacks
Post by: LoadWB on December 05, 2017, 04:18:33 PM
That's pretty neat.  Never used RequestChoice but I can see it being quite handy.
Title: Re: THORs Shell Hacks
Post by: Thomas Richter on December 05, 2017, 08:32:19 PM
Well, I thought about this issue. It is really a bit tricky, and there are more inconsistencies in the shell I want to get rid of.

First, we have some weirdo, namely variables with blanks in it. So if we have
Code: [Select]
set a=&quot;a b&quot;
Then, naturally,
Code: [Select]
command $a
results in "command" receiving two arguments, and
Code: [Select]
command "$a"
receiving one argument "a b".

If we set, however,
Code: [Select]
set a="*"a b*""
then
Code: [Select]
command $a
results in one argument, namely "a b". However, the V45 shell does for
Code: [Select]
command "$a"
also the same thing by first removing the outer quotes, hence giving effectively the same result. I believe this is wrong. As it is quoted, the quotes within the variable should be escaped, and hence the value preserved:
Code: [Select]
command "$a"
should call "command", but not with argument "a b", but with argument "*"a b*"", i.e. the quotes within the variable should be preserved.

As far as line feeds in variables are concerned: These can be really nasty. So if we have
Code: [Select]
set a="a*Nb"
and would then, without further modifications, call
Code: [Select]
command $a b
then the line feed within $a would terminate the line, and the command would never see the second argument. This is clearly undesired.

So I believe the following rule makes sense: If a variable is expanded within double quotes, it is expanded as a "binary variable", and the line feed stays within the double quotes, and is there - as part of the substitution process - replaced by a *N, so it is forwarded to the command.

If a variable is expanded outside of double quotes, the old rules apply and the variable is expanded as "text variable" and its contents is terminated at the first line feed, or \0, whatever comes first (this is how dos handles it).

I hope this makes sense...

A word about backticks: It is since ever that a line feed within backticks is replaced by a blank. This is also the case under Linux, and I believe it makes most sense in case a "list" command is put into the backticks. So I guess I keep it this way, unless somebody has a smart idea how to improve it without breaking too much.

Last but not least, we have one outsider, and this is the "rx" command. Unfortunately, this is not fixable as "rx" with arguments in quotes executes the script literally, i.e.
Code: [Select]
rx "say hello"
prints "HELLO" on the screen, whereas
Code: [Select]
rx say hello
executes the rexx script "say" (if it exists) with the argument "hello".

This all becomes a bit pathetic in case you have a volume named "say hello", as then the shell puts quotes around the volume name. Or to keep it simple, a rexx script on the "Ram Disk:". One cannot, possibly, execute it from there:
Code: [Select]
"Ram Disk:test.rexx"
on the shell will not work whatsoever, since the double quotes are needed to escape the volume name, but it changes the meaning of what "rx" attemts to do. Namely here not to execute a script on "Ram", but to execute the Rexx command "Ram" (which does not exist), with the argument "Disk:test.rexx". Bummer!

Replace "Ram Disk:" by "RAM:" and things work, but I guess Bill hasn't thought this one to the very end.

I hope there were people out here who could follow. This is all touchy business.
Title: Re: THORs Shell Hacks
Post by: LoadWB on December 06, 2017, 02:32:52 AM
I'm following.  I've had to deal with a lot of quirks regarding spaces and quotes in arguments and filenames.  Does not make me happy.
Title: Re: THORs Shell Hacks
Post by: olsen on December 06, 2017, 09:16:17 AM
Quote from: Thomas Richter;833827
Well, I thought about this issue. It is really a bit tricky, and there are more inconsistencies in the shell I want to get rid of.

Last but not least, we have one outsider, and this is the "rx" command. Unfortunately, this is not fixable as "rx" with arguments in quotes executes the script literally, i.e.
Code: [Select]

rx &quot;say hello&quot;
prints "HELLO" on the screen, whereas
Code: [Select]

rx say hello
executes the rexx script "say" (if it exists) with the argument "hello".

This all becomes a bit pathetic in case you have a volume named "say hello", as then the shell puts quotes around the volume name. Or to keep it simple, a rexx script on the "Ram Disk:". One cannot, possibly, execute it from there:
Code: [Select]

"Ram Disk:test.rexx"
on the shell will not work whatsoever, since the double quotes are needed to escape the volume name, but it changes the meaning of what "rx" attemts to do. Namely here not to execute a script on "Ram", but to execute the Rexx command "Ram" (which does not exist), with the argument "Disk:test.rexx". Bummer!

Replace "Ram Disk:" by "RAM:" and things work, but I guess Bill hasn't thought this one to the very end.

I hope there were people out here who could follow. This is all touchy business.


The ARexx shell commands might be fixable. I rewrote all of them in 'C' for AmigaOS4. The rx command is weird, though, but at least the 'C' language version uses ReadArgs() and falls back onto parsing GetArgStr() in case the "command line parameters" are a single line ARexx program.
Title: Re: THORs Shell Hacks
Post by: kolla on December 07, 2017, 01:28:34 AM
Quote from: LoadWB;833817
That's pretty neat.  Never used RequestChoice but I can see it being quite handy.


Yes it, plenty of room for more functionality though :)

As I mention, it center aligns, no matter, with fixed width fonts you can sort of work around it using some "invisble" character to have all lines equally long, but with normal fonts that gets messed up too. Would help if it understood tabs, for example.

Which reminds me - the shell has no concept of tabs, no *T :)

Btw, what I'm doing with this, is making a simple script with requesters for picking an available wifi network, parsing output from wpa_cli commands on "remote" (built into Amiga case) raspberry pi, via rsh (Thanks again, Olsen!). I try to avoid too much third party dependencies in my scripts, so they are more likely to work on all my systems, including the ones low on resources.
Title: Re: THORs Shell Hacks
Post by: Oldsmobile_Mike on December 07, 2017, 02:28:45 AM
Quote from: kolla;833786

(http://www.amiga.org/forums/picture.php?albumid=204&pictureid=1435)


Just quoting this because those are some of the most amusing requester choices I've ever seen.  ;)

Also, what are you using to get the additional buttons at the top of your Shell window?  (Help, Settings)
Title: Re: THORs Shell Hacks
Post by: kolla on December 07, 2017, 05:04:30 AM
That's just default ViNCEd config of OS3.9, I only set it to using ansi colours, which is why the background is black.
Title: Re: THORs Shell Hacks
Post by: Thomas Richter on December 07, 2017, 07:07:33 AM
Quote from: kolla;833866
Which reminds me - the shell has no concept of tabs, no *T :)
This is actually not a matter of the shell. While (parts of) the shell generate such funny sequences, it is the matter of ReadItem() of the dos.library (and -through that- ReadArgs() also in the dos.library) to interpret them. So the definition of additional "funny letters" is out of my hands (oh, wait, not exactly, but I do not want to touch dos this time).

Right now *T would just escape to T.

If you want to insert a TAB, well, on ViNCEd, type "ESC", then press TAB, and you get a TAB control sequence in the string. Or edit the shell skript with an editor of your choice. Tabs are "literals", they can be inserted directly into quotes and stay there.