Welcome, Guest. Please login or register.
Amiga Kit Amiga Store Iridium Banner AMIStore App Store A600 Memory

AuthorTopic: THOR's Shell Hacks  (Read 8843 times)

0 Members and 1 Guest are viewing this topic.

Offline Thomas Richter

Re: THORs Shell Hacks
« Reply #60 on: April 20, 2016, 07:35:26 PM »
Quote from: TuKo;807376
Dear Thomas,

I'm from the Apollo team and I'm following with attention your work on kickstarts modules.

As we are thinking to build a better ROM than we have today to give more functionnalities and bugfixes to users, I'd like to ask if you would kindly agree if we use your patches in our ROM ?

I mainly think :
* Shell 45.29
* layers.library 45.27

Thanks in advance

I have personally absolutely no problem with that, but you should also know that I'm not the owner nor full author of these modules. So I'm unfortunately not in the position of giving you (or denying you - which I wouldn't) access to these modules. So if in the end, hell breaks loose for whatever reason, my "allowance" helps nothing at all, and would be premature and even misleading.  

The shell-segment was done as part of Os 3.9, from where I currently derive my allowance to provide patches (but not full versions) of the shell. To my very knowledge, the IP of the Shell went back to Amiga, and from there to Hyperion, which also bought (to my knowledge) the rights on the 3.9 parts.  

Layers V45 was developed originally as part of Os 4.0, and hence was or is probably Hyperion's IP to begin with. At least the Os 4.xx code has a lot in common with the V45 layers as we have it today.

So in the end, I believe you need to talk to Hyperion to be really in a legally sound position, and you would *probably* need to make an arrangement with them to get a license to distribute the kickstart.  What I can do - if this is of any help - is to arrange a contact with them (and no, unlike what many believe, they don't eat children. Or poor developers for that matter.).  

I believe it's probably best if you'd approach me in private over my regular mail address. It hasn't changed over the years. You'll find it in the shell guide. Or elsewhere.  

  Please don't take this as a "no";. It really isn't - it's rather "I'll try my best to make it a yes".  

  I've a lot of respect what your team does, it's the best thing that happens to the platform in years. I'm simply not in the position of giving you something I do not own, and there are more people in the boat than you would probably imagine. Let's simply talk and I believe we'll find a way.
« Last Edit: April 20, 2016, 07:39:16 PM by Thomas Richter »
 

Offline TuKo

Re: THORs Shell Hacks
« Reply #61 on: April 30, 2016, 07:09:08 AM »
Dear Thomas,

I'd like to thank you for taking time to answer my request. I fully understand your point.

We are actually focusing on testing and being ready for the V500 which is coming soon so the topic for an upgraded kickstart is left a bit behind atm.

Cheers
 

Offline Thomas Richter

Re: THORs Shell Hacks
« Reply #62 on: January 12, 2017, 08:59:10 PM »
Probably in time to expand this tutorial a little bit by more "trivia". Did you know that you cannot redirect the output of "Execute" to a file?  

Thus, if you write a small script as follows
Code: [Select]
.KEY DIR

List <DIR>
save that as "T:myList" and then "Execute T:List", then - probably surprisingly - a

Code: [Select]
Execute T:myList SYS: >results

does not redirect the output of the script to "result". Instead, the directory listing just appears on the terminal. Weird?

Well, maybe, unless you know why that is. The point is that it isn't actually "Execute" that executes the script. Not at all. Execute just parses the script (incorrectly, actually, until recently), creates a temporary file in T:, and then leaves it to the shell to do the execution. So all the "gory work" is left to the shell. Execute is not doing very much at all.

By the time "Execute" returns, the execution of the script has not even started, but output redirection is un-done by the shell before the shell even attempts to read the temporary file in T:

The same goes of course for input redirection, or console redirection (the thing with "*>", see above).

Oh yes, at least until recently. I fixed that. Make "C:Execute" resident (ideally in the startup-sequence) , then open up a new shell, and then (and only then) input and output redirection works, even for Execute. That's more a workaround than a fix, given the above implementation, but at least one annoyance less.

By the way: The above script ("T:myList") contains at least potential problem. Do you see it? Can you fix it?

Probably more on this next time...
 

Offline kolla

Re: THORs Shell Hacks
« Reply #63 on: January 13, 2017, 12:12:11 PM »
Quote from: Thomas Richter;819623
By the way: The above script ("T:myList") contains at least potential problem. Do you see it? Can you fix it?

Not sure, please do tell :)

The way Execute parses the scripts can be used for funny exercises , such as changing .BRA and .KET conditionally throughout the script, the parsed script may very well look very different each time the script is executed.

One thing I have been missing, is a way to do "set echo on" in startup-sequence, but with output to a file in RAM: - for example if the echo variable could be a filename "set echo ram:bootlog"?
« Last Edit: January 13, 2017, 12:20:08 PM by kolla »
B5D6A1D019D5D45BCC56F4782AC220D8B3E2A6CC
 

Offline Kernel

Re: THORs Shell Hacks
« Reply #64 on: January 13, 2017, 04:50:58 PM »
Quote from: Thomas Richter;819623
Probably in time to expand this tutorial a little bit by more "trivia". Did you know that you cannot redirect the output of "Execute" to a file?  

Thus, if you write a small script as follows
Code: [Select]

.KEY DIR

List <DIR>

save that as "T:myList" and then "Execute T:List", then - probably surprisingly - a

Code: [Select]

Execute T:myList SYS: >results


By the way: The above script ("T:myList") contains at least potential problem. Do you see it? Can you fix it?

Probably more on this next time...


DIR is a command...?
 

Offline amoskodare

Re: THORs Shell Hacks
« Reply #65 on: January 13, 2017, 06:40:09 PM »
In that script DIR is the given name to a variable, which holds the parameter from Shell, which is passed on to the List command (if I'm not totally incorrect).
« Last Edit: January 13, 2017, 06:43:47 PM by amoskodare »
C128 + Action Replay, A500+ (KS1.3/KS2.0), A1200, A1200/040, Amiga Forever 2008+09+2016, Amiga Future subscriber, Nokia N900 (Maemo 5), 5 x86/x64 boxes
AmigaOS 4.1 FEu1 on Sam440ep-flex/800MHz/1GB RAM/Radeon 9250 :afro:
AOS4.1 FE Update 1 for Classic (on WinUAE PPC)
m4rko.com/AMIGA
 

Offline Thomas Richter

Re: THORs Shell Hacks
« Reply #66 on: January 13, 2017, 07:29:10 PM »
Quote from: kolla;819645
Not sure, please do tell :)
The problem appears as soon as you pass the script a parameter that contains spaces. So for example, try the following:

Code: [Select]
execute t:myList &quot;Ram Disk:&quot;
The problem is the following: Execute uses for parsing the arguments "ReadArgs()", similar to all other shell commands. Actually, it does so since Kickstart 1.2, or even since Tripos. (Note: ReadArgs(), even though it started to appear in Kickstart 2.0, is part of the dos "GlobVec" since the Tripos age, but was not exposed to AmigaOs before release 2.0. Anyhow...)

So, let's look at what ReadArgs does. It takes the entire argument, thus, "Ram Disk:" and places that in the argument list of Execute. Then, Execute puts that into "DIR". Hence, which value does "DIR" then have:
Code: [Select]
Ram Disk:
Obviously. Now, if that is inserted into the script, substituted for , we get the following temporary file:
Code: [Select]
List Ram Disk:
and this is what is passed to the shell. Do so see the problem?

Quotes are missing! Hence, what "List" sees are two arguments: "Ram" and "Disk:". And of course, "list" will error, because there is (likely) no file named "Ram" in the current directory, and no device or volume named "Disk:".

Bummer!

So, what can we do? Probably the following:
Code: [Select]
.key DIR

list "
"
So, just deliver the quotes, and everything is fine. "Ram Disk:" is again quoted, and a single argument, and if the file name is something without quotes, such as "SYS:", adding the quotes will not harm.

Really? Actually, not. And this is where the Execute V37.11, which is delivered with Os 3.1 stopped working and broke.

To see the problem, consider the following: You had enabled using the asterisk ("*") as wild card, so that
Code: [Select]
list *
will list all files.

Now try the follthe following:
Code: [Select]
execute t:myList *
and this won't work... "Unmatched quote", at least with the original "Execute", and even the one with 3.9. What is going on again?

Well, the original stupid execute replaces again the parameters with the arguments, and then left the following to interpret to the shell:
Code: [Select]
List "*"
The quotes are from your original quotes. If you do not see the problem with that remember that the asterisk is the BCPL escape-parameter, which only operates in quoted parameters - which we find here - and the combination *" means "a literal quote". And hence, the initial quote is not matched, and the shell errors.

Bummer! Similar problems can be constructed with file names like
Code: [Select]
Thisis"atest
or
Code: [Select]
AnotherStupidExample"
all of which are, surprisingly, completely valid and syntactical correct. A quote within a string is a literal quote, and does not quote anything (stupid, stupid shell syntax!)

If you quote that again, the literal quote within the strip becomes a functional quote, and messes the string up completely.....

Oh well - great job! Now, that's exactly what the new execute (V45.14) is good for. It really tries to understand what the string is, and escapes arguments correctly within quotes. With 45.14, the above example becomes
Code: [Select]
list "**"
instead. Execute notices that it has to substitute an argument within quotes, and hence has to escape the asterisk with another asterisk, and this - again - does the right thing. ReadArgs() will parse off the argument, will notice that it is quoted, will remove the quotes and re-interpret the "escape character" *, and replaces thus "**" by *. Which is what we want.

Similarly, the considerably wierd file name Thisis"atest becomes then
Code: [Select]
list "Thisis*"atest"
where *" is the escape sequence for the literal double quote.

Thus, Execute V45.14 learned how to parse the shell syntax, and place arguments back into quotes with proper escape characters in place. Which is also the reason why it is almost twice the size of the V37.11 version.

Hence, remember to quote execute parameters, and use a working execute.

(If this all causing a headache, you are not alone. I got a headache, too...)

Quote from: kolla;819645
The way Execute parses the scripts can be used for funny exercises , such as changing .BRA and .KET conditionally throughout the script, the parsed script may very well look very different each time the script is executed.
Indeed. I know the Bantam book and the manual actual gives examples like this, but whether that's all intended is the next question. I would really keep the "dot commands" to the start of the script and I would believe that this more works by accident than by design as it may actually backfire (as the quotes) to the script.

Now that the shell learned to understand "input file inclusion" with <

Quote from: kolla;819645
One thing I have been missing, is a way to do "set echo on" in startup-sequence, but with output to a file in RAM: - for example if the echo variable could be a filename "set echo ram:bootlog"?
Yes, right. One of the many old problems the shell has. "Echo" is pretty much legacy and probably ready for another update round. Though it's not exactly getting simpler....

I'd probably tell next time how the execute "Key" looks like. In the mean time, try to get a ".KEY" such as ".KEY BIRTHDAY/N" working. That, again, is another problem left over from Tripos. As in many other places.

So long,
Thomas
 

Offline kolla

Re: THORs Shell Hacks
« Reply #67 on: January 14, 2017, 05:53:19 AM »
Ah yes, paths with spaces, of course. Btw - I read the guide for your latest shell update, and the part about $STACK: it says it expects decimal as value, but surely you meant to write integer?
B5D6A1D019D5D45BCC56F4782AC220D8B3E2A6CC
 

Offline Thomas Richter

Re: THORs Shell Hacks
« Reply #68 on: January 14, 2017, 09:18:06 AM »
Quote from: kolla;819692
I read the guide for your latest shell update, and the part about $STACK: it says it expects decimal as value, but surely you meant to write integer?

Hmmm? It expects certainly an integer, but encoded in ASCII to the base of ten, i.e. a decimal number.
 

Offline kolla

Re: THORs Shell Hacks
« Reply #69 on: January 15, 2017, 01:22:14 PM »
Quote from: Thomas Richter;819696
Hmmm? It expects certainly an integer, but encoded in ASCII to the base of ten, i.e. a decimal number.

Yeah, well - but decimal numbers may contain fractions, for example 11.37 :)

No big deal though, just confused me a little.
« Last Edit: January 15, 2017, 01:25:52 PM by kolla »
B5D6A1D019D5D45BCC56F4782AC220D8B3E2A6CC
 

Offline kolla

Re: THORs Shell Hacks
« Reply #70 on: January 15, 2017, 01:39:46 PM »
No really related to the shell itself, but anyhow... Something I have pondered on what/what defines LFORMAT entities for c:List, for example? And why isn't there one that is like %P, only without trailing / ? :) Nor is there an option to list a drawer itself instead of the content of a drawer - right? I am missing the equivalent of *ix "ls -d".

Also it would be nice to override locales, just today I was taken off by how I have to use norwegian like
Code: [Select]
list all files since &quot;i dag&quot; ("i dag" meaning "today") - I don't mind that I _can_ use norwegian, but it would for sure be nice, for scripts etc, that english worked as well. Same for input to C:Date.
B5D6A1D019D5D45BCC56F4782AC220D8B3E2A6CC
 

Offline Thomas Richter

Re: THORs Shell Hacks
« Reply #71 on: January 15, 2017, 02:17:18 PM »
Quote from: kolla;819768
Something I have pondered on what/what defines LFORMAT entities for c:List, for example? And why isn't there one that is like %P, only without trailing / ? :) Nor is there an option to list a drawer itself instead of the content of a drawer - right? I am missing the equivalent of *ix "ls -d".
This is a particular shortcoming, and probably due "list" just using the arp pattern matcher (now part of dos.library). If you think about it, "LFORMAT" is neither bullet proof.

So, for example, what happens if your directory contains a file whose file name includes a quote or an asterisk? While I have not checked, I would *bet* that "list LFORMAT" forgets to escape such characters properly, i.e. a file name including a quote with an LFORMAT that includes its argument in quotes as well should replace " by *" and * by **.

It is really astonishing how little thought the creators have put into the syntax of the shell.

I don't think "list" can be as easily replaced as "execute" due to its many options.

Quote from: kolla;819768
Also it would be nice to override locales, just today I was taken off by how I have to use norwegian like
Code: [Select]
list all files since &quot;i dag&quot; ("i dag" meaning "today") - I don't mind that I _can_ use norwegian, but it would for sure be nice, for scripts etc, that english worked as well.

It would not be only "nice", it is a necessity to allow portable scripts that work independently of the locale.

As you note correctly, there are lots of problems with the shell, and it will take a while to clean up all the mess. It seems I've just scratched the surface.
 

Offline Thomas Richter

Re: THORs Shell Hacks
« Reply #72 on: February 01, 2017, 07:59:27 PM »
Back to "execute" - as announced a while ago. As you probably know, shell scripts may take arguments, and quite like "real" shell commands, shell scripts also use a "template" that defines the type of the arguments, whether they are required or optional, whether they are switches, toggles and similar things. That's what the ".key" pseudo-command on the first line of a script does:

Code: [Select]
.KEY FROM/A/M,TO/A,CLONE/S

copy <FROM> <TO> <CLONE>
would be a simple wrapper to the "copy" command, where "FROM" is a mandatory argument, and multiple of them can be specified. "/A" means "mandatory", and "/M" means "one or more of them". "TO" is also a required argument, and "CLONE" is a switch.

Identical to binary commands, the letter behind the slash defines the type and requirements of the argument, and thus
Code: [Select]
.KEY YEAR/N

echo "We are in year "
would take a numeric argument, and refuse anything that is not a number.

That is how it is documented, and how arguments work. Execute uses the dos.library "ReadArgs()" function to parse off arguments, so all the nice "/" types come for free.

Unfortunately, however, none of the above examples work if you try them with the original v37 "Execute" command, and that is the one that came with workbench 3.1. Now, how come? Didn't I just say that "Execute" uses the dos.library "ReadArgs()", and that for this reason argument parsing should just work, because "ReadArgs()" surely works? Why doesn't it work for "Execute" then?

Well, this requires a bit of background. What ReadArgs() does is that it fills an array of string pointers, one for each argument. Or at least, this is what "rdargs()", the BCPL counterpart in Tripos did back then in the Kickstart 1.3 times. So the BCPL "Execute" from 1.3 did just work.

The problem is that kickstart 2.0 added a couple of new argument types. So for example, "/N" for "numeric argument" is new, and "/M" for "multiple arguments" is new. It did not exist in 1.3, and these two types were added by "arp", the amiga replacement project, and became later part of AmigaDos.

What "arp" also did is that it changed a bit the convention of how "ReadArgs()" works. So for example, for a "/N" argument, it is of course much more convenient to return the number directly instead of a string, so "ReadArgs()" fills in a pointer to a LONG instead of a pointer to a character array. This is certainly much more convenient than having to parse the number *again* in the program - ReadArgs() can do that already.

Similarly, for "/M" ReadArgs() does not leave a string pointer in its return vector, but rather, returns a pointer to a string array, i.e. a pointer to a pointer of characters. It has to, since "/M" may take multiple arguments on the console, so one string is not enough.

Unfortunately, nobody told "Execute" on the changed syntax, and nobody dared to touch the execute sources either. Thus, if the ".KEY" contains a "/N" argument, the v37 execute (the official 3.1 one, note that there is no v40 execute) expects a string pointer in the arguments, but just finds a pointer to a number there. And does strange and wonderful things, of which an "Enforcer hit" is probably the most harmless. Outch!

Hence, someone (the software distillery, namely) changed the interface of an Os function behind the "back of Execute", and thus broke a command. Or rather, they forgot to update Execute.

Unfortunately, this change means that now execute also needs to be aware of the type of arguments it has to parse off, and has to convert the numbers or multiple strings back to one single string, namely those that it requires for substitution of the parameters of the script, i.e. the things enclosed in angle brackets.

Hence, for V45 execute (and also V44), a couple of things unfortunately happen twice: First, ReadArgs() converts strings to numbers for /N arguments (because it always does), and "Execute" has to convert them back (for parameter substitution) such that the command in the script has to convert them back from string to number again.

There are more types like this: "/S" just leaves a boolean flag instead of a string, so "Execute" has to re-insert the name of the switch instead of the boolean value. The same goes for "/T", which is actually not really a toggle, but takes two values, "ON" or "OFF", or alternatively "YES" or "NO". The documentation is not quite appropriate.

"/M" returns a pointer to a string array, so "Execute" has to merge all the strings together.

And last but not least, do not forget that Execute also has to take care of proper quoting arguments with blanks in it - and escaping of the asterisk and the quote in the strings itself. We had this before.

So, it's actually a bit more complicated than it seems.
 

Offline kolla

Re: THORs Shell Hacks
« Reply #73 on: February 02, 2017, 12:53:36 PM »
Can you say something about when Execute is used, and hence what shell features one can use in the different cases? For example, it is not used to execute S:Startup-Sequence, but it is, by default used for S:User-Startup. How about S:Shell-Startup? :)
B5D6A1D019D5D45BCC56F4782AC220D8B3E2A6CC
 

Offline Thomas Richter

Re: THORs Shell Hacks
« Reply #74 on: February 02, 2017, 03:37:16 PM »
Quote from: kolla;821345
Can you say something about when Execute is used, and hence what shell features one can use in the different cases? For example, it is not used to execute S:Startup-Sequence, but it is, by default used for S:User-Startup. How about S:Shell-Startup? :)

When a program calls "SystemTagList()" or "Execute()" (the dos.library call), then the "Execute command" will *not* be used (ok, unless of course it is a command in the script being executed - clearly). The commands will just go to the shell directly as "command input". This happens for the startup-sequence, and also for s:shell-startup and s:cli-startup. In the first case, it is the dos.library that (implicitly, not directly) initiates the bootstrap shell. In the two other cases, it is the Sys:System/CLI binary that simply calls "Execute()" (the dos.library function call, not the command)".

The "Execute" command *is* used if you either call it explicitly (clearly, as for the "User-Startup") or if the S-bit of a script is set and the shell identifies it as shell script.

Even in the latter case, one cannot really say that "Execute executes the script". Not at all. At best, it performs argument substitution, and then just "bends a couple of pointers" of the current CLI to leave all the dirty work to the shell.

"Execute" is really a confusing name for this binary. It's more the "CLI command input stream redirection" binary, but that would have been too long a name, I suppose. A better command name would have been "<". If you understand this joke, you got the shell syntax and the idea how execute works.