Welcome, Guest. Please login or register.

Author Topic: THOR's Shell Hacks  (Read 34957 times)

Description:

0 Members and 1 Guest are viewing this topic.

guest11527

  • Guest
Re: THORs Shell Hacks
« Reply #29 on: February 09, 2015, 02:06:52 PM »
Quote from: olsen;783396
The major reasons for starting over were in portability (ViNCed was written in 68k assembly language, and would have had to be reimplemented), and the unresolved limitations of the "shell" Kickstart module which, for example, in Kickstart versions 2.04 through 3.1 had a fixed upper limit for the length of an individual command line (the OS4 version has no fixed upper limit).

For reference, the 3.9 version does not have an upper limit either. It uses its own string buffering mechanism and self-expanding buffers, i.e. whenever you use a command longer than the current buffer, a new buffer of the necessary size is created. The 3.1 shell had a maximum command length of 512 bytes.

I wonder what happened with all the other shell bugs (e.g. the one related to "skip back" I mentioned above).
 

Offline ChaosLord

  • Hero Member
  • *****
  • Join Date: Nov 2003
  • Posts: 2608
    • Show only replies by ChaosLord
    • http://totalchaoseng.dbv.pl/news.php
Re: THORs Shell Hacks
« Reply #30 on: February 09, 2015, 04:28:36 PM »
I would like to officially say:

Holy Crap!

I never knew any of this stuff!

I run OS3.9 but mine did not come with any instruction manual.

Maybe there was one on the CD that I never looked at?

The last manual I studied came with my A4000 OS3.0.  If I forget something then that is the manual I autograb off the shelf.  I guess that is a pretty stupid thing to do in 2005+ since there must be a better AmigaDOS manual somewhere...
Wanna try a wonderfull strategy game with lots of handdrawn anims,
Magic Spells and Monsters, Incredible playability and lastability,
English speech, etc. Total Chaos AGA
 

Offline pVC

Re: THORs Shell Hacks
« Reply #31 on: February 09, 2015, 04:52:41 PM »
Quote from: ChaosLord;783404

I run OS3.9 but mine did not come with any instruction manual.

Maybe there was one on the CD that I never looked at?

The last manual I studied came with my A4000 OS3.0.  If I forget something then that is the manual I autograb off the shelf.  I guess that is a pretty stupid thing to do in 2005+ since there must be a better AmigaDOS manual somewhere...


Doh, yes, there is online manual on the CD. I've copied it to HD to check it always when needed. It's basically structured the same way like the printed 3.0/3.1 manuals and splitted to few separate books. It's really handy to have it available all the time on your machine.
Daily MorphOS user and Amiga active.
 

Offline lionstorm

Re: THORs Shell Hacks
« Reply #32 on: February 09, 2015, 05:46:03 PM »
Quote from: ChaosLord;783404
I would like to officially say:

Holy Crap!

I never knew any of this stuff!

I run OS3.9 but mine did not come with any instruction manual.

Maybe there was one on the CD that I never looked at?

The last manual I studied came with my A4000 OS3.0.  If I forget something then that is the manual I autograb off the shelf.  I guess that is a pretty stupid thing to do in 2005+ since there must be a better AmigaDOS manual somewhere...


a nice chap here at a.org scanned them and converted them into amigaguide and pdf : http://www.amigamanuals.co.uk/
 

guest11527

  • Guest
Re: THORs Shell Hacks
« Reply #33 on: February 09, 2015, 05:47:51 PM »
Quote from: ChaosLord;783404
Maybe there was one on the CD that I never looked at?

There should be a shell.guide (if not there then in Aminet with the shell update) which contains most of the information shown here (except the "Skip lab" episode), and there is the ViNCEd.guide (also in Aminet, part of the ViNCEd distribution) which explains all the above.
 

Offline Thorham

  • Hero Member
  • *****
  • Join Date: Oct 2009
  • Posts: 1149
    • Show only replies by Thorham
Re: THORs Shell Hacks
« Reply #34 on: February 09, 2015, 05:54:42 PM »
To Thomas Richter:

Any chance of making the source code of ViNCEd available?
 

Offline kolla

Re: THORs Shell Hacks
« Reply #35 on: February 09, 2015, 10:12:57 PM »
Personally I find ViNCEd a bit "too much!!" and wish that job control, tab expandation etc could be moved to the shell itself.
B5D6A1D019D5D45BCC56F4782AC220D8B3E2A6CC
---
A3000/060CSPPC+CVPPC/128MB + 256MB BigRAM/Deneb USB
A4000/CS060/Mediator4000Di/Voodoo5/128MB
A1200/Blz1260/IndyAGA/192MB
A1200/Blz1260/64MB
A1200/Blz1230III/32MB
A1200/ACA1221
A600/V600v2/Subway USB
A600/Apollo630/32MB
A600/A6095
CD32/SX32/32MB/Plipbox
CD32/TF328
A500/V500v2
A500/MTec520
CDTV
MiSTer, MiST, FleaFPGAs and original Minimig
Peg1, SAM440 and Mac minis with MorphOS
 

Offline ChaosLord

  • Hero Member
  • *****
  • Join Date: Nov 2003
  • Posts: 2608
    • Show only replies by ChaosLord
    • http://totalchaoseng.dbv.pl/news.php
Re: THORs Shell Hacks
« Reply #36 on: February 09, 2015, 10:38:28 PM »
@Thor

Did you program the OS3.9 Shell to use a static memory model?
(That is a memory usage that stays static 99% of the time after it has been started)

I know it will change the amount of ram used if I use a shell command >512 chars but other than that does it go around do malloc();free(); here and there?


The reason I ask is because I do all my compiling and testing from shell.  I test for memory leaks a lot.  Many years ago I found that all the fancy shells that I tested such as KingCON: keep randomly changing the amount of ram that they consume so I can't easily check for memory leaks in my programs.
I think its because of their scrollback buffer constantly expanding.

I like to do
Code: [Select]

avail flush
run myGiantComplicatedProgram blahblah blah moreblah
avail flush

to see if my memory usage is the same before & after.

But that did not work in KingCON: and whatever fancy shells I tested.

I have a vague memory that I may have downgraded my shell back to the old OS3.1 shell many years ago.  I just can't remember exactly.  My history file with all my documented tests was lost in a great hard drive cataclysm :(

I always knew you were a legendary megacoder who does not merely write code but SMASHES code into existence with his mighty magic hammer (+40 vs. bugs) but I had no idea you coded the 3.9 shell.
Wanna try a wonderfull strategy game with lots of handdrawn anims,
Magic Spells and Monsters, Incredible playability and lastability,
English speech, etc. Total Chaos AGA
 

Offline ChaosLord

  • Hero Member
  • *****
  • Join Date: Nov 2003
  • Posts: 2608
    • Show only replies by ChaosLord
    • http://totalchaoseng.dbv.pl/news.php
Re: THOR's Shell Hacks: CONSOLE jobs.
« Reply #37 on: February 09, 2015, 10:51:12 PM »
Quote from: Thomas Richter;783399

Now, if "console:" addresses a named console, what does "console:" stand for? It is the "startup console", i.e. the console that was started along with the window. And what is "*" for? It is the "current console", namely the console that is currently active and in foreground.

Thus, "CONSOLE:" and "*" are not necessarily identical, or rather, they are only identical if the current console is the console that was created along with the window - or the only console.

Thus, it is typically safe to output to "*", as this is in foreground always, and it is not always safe to output to "CONSOLE:" as this console might be in background, and you might get suspended.

Thus, "CONSOLE:" != "*", and the file name behind "CONSOLE:" has a specific meaning, it identifies one out of several consoles that share the same window as output. All the job control and output control is done by ViNCEd, and you can switch between the consoles with the SetVNC program or tiny scripts in S:.


So for people who are using CLI (that old ancient thing) or OS3.1 Shell it would be accurate to say that for those users * = CONSOLE:  True?

But for people using (OS3.9 shell) or (OS3.1 + ViNCEd)
* <> CONSOLE:

I am just bringing this up because somewhere out there, there is a bug being caused by some script assuming that * = CONSOLE:
Wanna try a wonderfull strategy game with lots of handdrawn anims,
Magic Spells and Monsters, Incredible playability and lastability,
English speech, etc. Total Chaos AGA
 

Offline klx300r

  • Amiga 1000+AmigaOne X1000
  • Hero Member
  • *****
  • Join Date: Sep 2007
  • Posts: 3244
  • Country: ca
  • Thanked: 20 times
  • Gender: Male
    • Show only replies by klx300r
    • http://mancave-ramblings.blogspot.ca/
Re: THORs Shell Hacks
« Reply #38 on: February 10, 2015, 03:03:35 AM »
thanks Thomas ! Love this very useful thread:)
____________________________________________________________________
c64-dual sids, A1000, A1200-060@50, A4000-CSMKIII
Indivision AGA & Catweasel MK4+= Amazing
! My Master Miggies-Amiga 1000 & AmigaOne X1000 !
--- www.mancave-ramblings.blogspot.ca ---
  -AspireOS.com & Amikit- Amiga for your netbook-
***X1000- I BELIEVE *** :angel:
 

Offline kolla

Re: THOR's Shell Hacks: CONSOLE jobs.
« Reply #39 on: February 10, 2015, 04:21:55 AM »
A posting about SPARWED, ie the protection flags, and how H (hold) now works again could be nice, as people seem ignorant about them.
B5D6A1D019D5D45BCC56F4782AC220D8B3E2A6CC
---
A3000/060CSPPC+CVPPC/128MB + 256MB BigRAM/Deneb USB
A4000/CS060/Mediator4000Di/Voodoo5/128MB
A1200/Blz1260/IndyAGA/192MB
A1200/Blz1260/64MB
A1200/Blz1230III/32MB
A1200/ACA1221
A600/V600v2/Subway USB
A600/Apollo630/32MB
A600/A6095
CD32/SX32/32MB/Plipbox
CD32/TF328
A500/V500v2
A500/MTec520
CDTV
MiSTer, MiST, FleaFPGAs and original Minimig
Peg1, SAM440 and Mac minis with MorphOS
 

guest11527

  • Guest
Re: THORs Shell Hacks
« Reply #40 on: February 10, 2015, 06:00:00 AM »
Quote from: kolla;783429
Personally I find ViNCEd a bit "too much!!" and wish that job control, tab expandation etc could be moved to the shell itself.

So do I, and ViNCed is even prepared for that. Just that I had never found the time to move this code to the shell. As far as ViNCed is concerned, it would not require a code change. Well, except that I could retire a lot of code with such shell upgrades in effect. More on this later.
 

guest11527

  • Guest
Re: THOR's Shell Hacks: CONSOLE jobs.
« Reply #41 on: February 10, 2015, 06:02:44 AM »
Quote from: ChaosLord;783438
So for people who are using CLI (that old ancient thing) or OS3.1 Shell it would be accurate to say that for those users * = CONSOLE:  True?
As long as you stick to CON:, yes. As said, it is really up to the console to interpret the "file name", so one cannot generalize this. I believe AUX: also handles * and CONSOLE: alike and ignores the file name, which is the second console (the serial one) AmigaOs has.
 

guest11527

  • Guest
Re: THORs Shell Hacks
« Reply #42 on: February 10, 2015, 06:23:15 AM »
Quote from: ChaosLord;783434
@Thor

Did you program the OS3.9 Shell to use a static memory model?
No, and for several reasons. You already pointed one out, and that's the dynamic command line length - which can grow arbitrarely large with the 3.9 shell. There are many others. For example, the shell also allocates the stack for the commands it allocates, it allocates memory (indirectly) by loading binaries and opening file handles, and it also creates temporaries in memory for expanding backticks (more on this later) or for the input
 

guest11527

  • Guest
Re: THORs Shell Hacks
« Reply #43 on: February 10, 2015, 11:38:39 AM »
Ok, let's look today at how the shell executes commands. Sounds boring? You type the command, the shell locates it in the path, and executes it, right?

Well, not quite so. There are a couple of catches. First, there is the "+" sign at the end of the line. It is often misunderstood to continue the current command line at the next line, but then why

Code: [Select]
7.SCSI:> echo hello +
world
does only print "hello" and not "hello world"? On the other hand,

Code: [Select]
7.SCSI:> run +
list
runs the list command, indeed. So what's the difference between "echo" and "run"?

Well, the plus sign also inserts a line feed into the command line, and "echo" simply stops parsing its command arguments there. The rest of the string remains in the standard input of the command, at least if it uses buffered I/O, and waits there to be retrieved after parsing. "Run" parses multiple lines, but all other commands simply stop at the first line feed.

Anyhow, to return to the "+", the following demonstrates this:
Code: [Select]
7.SCSI:> ask "Please enter a number:" numeric to x
The shell will now kindly ask you to enter the number, so do it the pleasure:
Code: [Select]
Please enter a number: 42
Did this number arrive?

Code: [Select]
7.SCSI> echo $x
42
Yes. Fine so. But we can also feed the answer right into "ask" by using this little trick:

Code: [Select]
7.SCSI> ask "What do you get if you multiply six by nine?" numeric to x +
54
At this point, ask doesn't even wait for you to respond, but takes the 54 from the next line and pushes it into its input. Thus, the ask command returns immediately.

There is another magic token like "+", namely "&". This works like "run", actually very much like it, as it executes a command in the background:

Code: [Select]
7.SCSI> list &
7.SCSI>
[CLI 8]  list suspended. [ViNCEd output]
7.SCSI>
Thus, the list command is started, run in background, but as it tries to write something to console, but is not the current console owner (see yesterday's episode), it is suspended. To continue, place it in foreground:

Code: [Select]
7.SCSI> fg 8
If you simply want to run a command in background, redirect its input and output to NIL: Of course, this does not make much sense with list, but anyhow:
Code: [Select]
7.SCSI> list <>NIL: &
Starts list in background, and returns to the shell.

Now, besides commands, we also have scripts. A script is a file whose "s" protection bit is set; these are typically found in S:, for example the "fg" command is one.

If the S-bit is set, the shell does something else. Usually, you would say "execute this as a shell script", but hold on, it's not quite that simple.

It first tries to determine the type of the script. If the first two characters of the file are "/*", i.e. forwards slash, star, then the file name is taken as is, a "rx" is prepended at its start, and the result is executed. Hence, execution is not done by the shell, but by ARexx. These two "magic characters" identify rexx scripts rather than shell scripts.

There are more "magic characters" than /*. These are #! and ;!, both work identical. If a file whose s-bit is set has them as first characters in the first line, the rest of the line denotes the name of a command to execute to run the script, and the rest of the command line would be given to this command. For example, if Amiga had a pyhton interpreter, the initial line "#! C:python" would run the python interpreter. But we can try something similar:

Code: [Select]
7.SCSI:> echo ";! C:ed" >ram:test
7.SCSI:> protect ram:test s add
7.SCSI:> ram:test
This makes the shell believe that "ram:test" is a script whose interpreter is the editor Ed, and by executing it, it gets run with the name of the "script to execute", namely "ram:test", as first argument.

Now, that's the S bit. But there is more. You know of course the feature that the shell implicitly executes a "CD" - change directory - if you just type in the name of a directory or device:

Code: [Select]
7.SCSI:> ram:
7.Ram Disk:>
But what does it do in case the file is not executable and not a directory? Well, even then the shell can do something, though it depends on setting the right variables. So let's help the shell a bit:

Code: [Select]
7.Ram Disk:> set VIEWER SYS:Utilities/Multiview
7.Ram Disk:> Guides:Shell.guide
Now, for that to work, I assume that you have "Shell.guide" in an assign named "Guides:", and that the "e"-bit, for "executable", of this guide is *NOT* set. If it is set, the shell will try to execute the "Shell.guide" as binary, and that of course gives you an "object is not executable".

The way how this is arranged here, however, the shell will determine that the input is a datatype, and will run this datatype through the "viewer", which is "Multiview".  This "VIEWER" variable setting can be made permanent in ENVARC: if you like:
Code: [Select]
7.Ram Disk:> echo "Sys:Utilities/Multiview" >ENVARC:Viewer
and if the above example does not work, remove the "e" bit:
Code: [Select]
7.Ram Disk:> protect guides:Shell.guide e sub
Now, one final "feature" of the shell. Assume that you have the mmu.library in libs:
Code: [Select]
7.SCSI:> list LIBS:mmu.library
mmu.library                       56952 ----rwed 30-May-14 12:01:44
1 file - 113 blocks used
7.SCSI:>
What does the following do?

Code: [Select]
7.SCSI:> LIBS:mmu.library
It prints mmu.library: Unknown command", even though the "command" is right in front of the face of the shell, and it is even an AmigaOs binary, and it even has its "e" bit set, so it should be executable. Why doesn't the shell then try to execute the thing?

The fact is, it actually does. However, libraries and devices are not loaded into the system like this, at least not normally (vnc.library is an exception, hear, hear). So what the "startup code" of the library does is to simply return a "-1" as return code to the shell. And this magic return code, undocumented to my very knowledge, tells the shell "hey, I'm actually not an executable, print an error to the user". Even though it is actually executed, though doesn't do anything useful.
 

guest11527

  • Guest
Re: Variable Shell Hacks
« Reply #44 from previous page: February 11, 2015, 12:51:32 PM »
How much do you know about shell variables? These work like in any other programming language, REXX for example. You assign them a value like this

Code: [Select]
7.SCSI:> set foo bar

and you retrieve their value a little bit different, namely by putting a dollar sign in front of their name:
Code: [Select]
7.SCSI:> echo $foo
bar

Where do these variables come from? Well, from several places. First, if you just use "set" as above, they come from a shell internal database. This is different for each shell, and hence, such variables are "local" to the shell you use them.

A second type of variables are "global environment variables". They are set with "setenv" instead of "set", but work otherwise the same way:

Code: [Select]
7.SCSI:> setenv hi bar

and you retrieve their value a little bit different, namely by putting a dollar sign in front of their name:

Code: [Select]
7.SCSI:> echo $hi
bar

If two variables of the same name exists, i.e. one environment variable and a local shell variable, the latter is used. Actually, environment variables exist as a file in ENV:, an assign to a directory in the RAM-disk, which is usually copied from ENVARC: during startup. Thus, you can make environment variables permanent by copying their file from ENV: to ENVARC:

Or you can do that in one step:

Code: [Select]
7.SCSI:> setenv save hi bar

the "save" keyword also makes the change permanent and places the variable file in ENVARC: and in ENV:, so it will survive a reboot.

Now, there is a little bit more that can be said about variables? What do we do if we want to write "$foo" on the console, as a literal string, and do not mean the variable? Well, use the BCPL escape-character, which is the asterisk:

Code: [Select]
7.SCSI:> echo "The value of *$foo is $foo"
The value of $foo is bar

Now, here comes the wierd part: Didn't I say that the "*" works only as escape character within quotes? Yes, but... for *$ this is different, it works always:

Code: [Select]
7.SCSI:> echo *$foo
$foo

Why this inconsistency? Well, the original BPCL shell was drilled up over and over again, and apparently, CBM didn't quite know what they were doing, so consistency was apparently not an option, or the inconsistency was not recognized. In the old shell code, parsing and expansion of variables and parsing of strings and quotes was handled in two separate steps, so the code did not even know whether the $ was inside or outside of a double quote.

Speaking of double quotes, did you know that a double quote *within* a string is not understood as a double quote?

Code: [Select]
7.SCSI:> echo thisisa"doublequote
thisisa"doublequote

If there would have been a space in front of the quote, it would have acted as a quote, but this one did not. Every sane programming language would have created an "unterminated string" error for the above line. AmigaDos does not. A quote in the middle of a string is not a quote. Why? Nobody knows...

Back to variables: If you want other characters such as spaces in the variable name, or want to make sure that the variable name is separated from the string, you may enclude the name in curly braces:
Code: [Select]
7.SCSI:> echo ${foo}bar
barbar
without the curly braces, the shell would have tried to expand the variable foobar, which does not exist. If a variable does not exist, it just expands to its name, so let's try this:
Code: [Select]
7.SCSI:> echo $foobar
$foobar
Now, how can we know whether a variable exists? That's what $? is good for:
Code: [Select]
7.SCSI:> echo $?foo
1
7.SCSI:> echo $?foobar
0
7.SCSI:> echo $?{foo}bar
1bar
So $?var expands to 1 if the variable exists, and 0 otherwise, and the last example just showed that curly braces work with $? as well, to separate the variable name "foo" from the rest of the string "bar".

Tomorrow: More on variables. Or actually, things that look like variables, but really aren't. The "alias" command introduces something that works like a variable, but isn't. Again, only CBM knowns why they made this so inconsistent. And then, scripts can take arguments, filled in by "execute". These also look like variables, but strangely enough, they aren't. What CBM was smoking at this time, nobody knows, but it's really another completely different mechanism.