Welcome, Guest. Please login or register.
Amiga Kit Amiga Store Iridium Banner AMIStore App Store A600 Memory

AuthorTopic: The Os 3.1.4 Thread  (Read 51967 times)

0 Members and 2 Guests are viewing this topic.

Offline kamelito

Re: The Os 3.1.4 Thread
« Reply #495 on: July 09, 2019, 07:14:40 AM »
@Thomas Richter thanks for the details.
Not sure if this is a bug but I’ve seen this related to AmigaOS 3.1.4.
http://eab.abime.net/showthread.php?t=98038
 

Offline Thomas Richter

Re: The Os 3.1.4 Thread
« Reply #496 on: July 09, 2019, 02:37:52 PM »
Well, probably time to continue with another episode. This time with my "most beloved component", which is HDToolBox.

First, one bug I am responsible for: If you scan a SCSI chain, and there is a "hole" in the assigned IDs - e.g. SCSI IDs 0 and 1 are present, 2 is not, but 5 and 6 are - then HDToolBox stopped looking for any further drives once it found a non-existing ID. The story is that I was actually attempting to abort the loop for scanning for LUNs, logical units, but I hit the wrong loop. Oh well,  this is fixed.

All other issues were old, from 3.1 times and before. The first newly discovered issue is how HDtoolbox "determines" the drive geometry, i.e .what you see as "sectors per block" "number of tracks" and so on. Nothing of that data is in good relation to reality, and most of the data is just "phantasy" the HDToolBox makes up. Background is that SCSI disks simply address disks as a linear list of sectors, and even if they have tracks, the track size varies by the cylinder, so the physical reality does not fit into the "rigid" scheme used by the dos.library for mounting disks. HDToolBox has just to make up a reasonable geometry such that the entire number of sectors fits, or is at least not larger than the real number of sectors.

Problem is, what is "reasonable". Well, if you had a disk initialized, HDToolBox reserved two tracks for the RDB to store the file system in. Always. Unfortunately, if the disk geometry is not reasonable, two tracks may be just "very little", e.g. if the algorithm generated a disk geometry (all made up, as said) with 2 sectors per track. Then there would be a total number of four sectors available for the RDB - where it does not fit in, of course.

So that got fixed, both the algorithm how the disk geometry is estimated, or made up, and second, the number of tracks to reserve for the RDB.

Then, we had a couple of issues where users put the FFS on a quite large (for Amiga times, at least) partition, and then received an "Out of memory" when attempting to validate the partition. This can be resolved by using a larger block size of 4096 bytes rather than 512 bytes, and the former is recommended for many modern drives and SD cards anyhow that work internally with 4K sectors. So HDToolBox now switches to 4K sectors when the partition size is large enough, and it also aligns partitions to 4K boundaries.

Finally, we have two "freak accidents" in the HDToolBox, errors that are real "face palm" experiences. No, this time not caused by me. First, the stack size of a file system. HDToolBox was kind enough to recognize that the FastFilesystem is not written in BCPL and hence does not require a "GlobVec", but it did not enter a stack size for it, so the stack size remained at its default, as created by the expansion.library when creating the mount list of the drive. The default is 600.

The right question is now "600 what"? And the answer is "it depends...". Unfortunately, the dos.library knows two types of handlers: BCPL handlers and C/assembly handlers. The former are created by an internal function named "createtask" (not to be confused with exec tasks) and it takes the stack size in long words. The latter are created by "CreateProc", which takes the stack size in bytes. So, if you mount a handler with "GlobVec=0", then the stack size in the mount list means "long words", and if it is a C mount, the stack size means "bytes". Unfortunately, the default left behind by the expansion library is "BCPL mount", which - with a stack size of 600 - results in 2400 bytes of stack. As soon as the HDToolBox patches this to "this is a C mount", the result is a stack size of 600 *bytes*, which is really quite tight, even for the FFS.

So, in the end, HDToolBox now also leaves sufficient information to extend the stack size to 2048 bytes.

The second accident is in the bytes/block settings for the fast file system. Betatesters reported that whenever a file sytem was removed from the RDB, the block size was reset to 512 bytes, which then resulted in an unsuable partition if the partition was originally formatted with larger block sizes. This sounds like a (well not harmless but) simple GUI bug, but it wasn't. To understand the issue, you need to know how partitions are mounted.

Step 1) is that the expansion.library creates a "template" for the mount list (or its equivalent), which is "globvec = 0" (BCPL mount), 600 LONGs stack (see above), 1 sector per block.
Step 2) is that the firmware of the hostadapter uses this template and fills in the partition boundaries from the disk, such as "LowCyl" and "HighCyl".
Step 3) is that the file system is loaded from RDB and has a couple of ideas of its own what it needs, so for example to get the GlobVec and the stack size replaced. This is communicated by "PatchFlags" which instruct the firmware to replace some of the entries created in 1) or 2) by data by its own.

Now, if you instruct HDToolBox to use a different number of sectors per block, two things happen: First, the partition layout is modified, that is, the data in step 2). Second - and this is the crazy part - HDToolBox also updates the Patchflags for the file system with something that looks "apparently" as if the block size is also to be modifed at file system level. (1<<DE_SECTORSPERBLOCK) is or'd into the patch flags.

This, however, would be fatal: A file system with the sectors per block patched over would mean that the sectors per block from the partiion is replaced. So if you create two partitions, one with 512 byte blocks, and another with 4096 bytes per block, and use one file system for both of them, the above patch flags seem to override the block size for both - and then ruins at least one of the partition.

Luckely (ehem) another bug compensates the first bug: (1 << DE_SECTORSPERBLOCK) is actually the wrong flag (face palm #2). Due to the way how the patch flags are enumerated, it modifes the stack size. Harmless for the partition, but leaves the stack size then at 0. Ok, the dos.library is not quite as stupid to allow this size, and then selects one by itself.

What is left is a typical exercise in database management: If you have two data entries that mean the same (block size at partition level, block size at file system level), you need to keep them consistent. HDToolBox attempted to do so, but failed. Whenver a file system was deleted, one part of the information was removed, and then the other - at partition level - was removed as well, resulting in an invalid block size at partition level.

So, two bugs, with only one of them the result might be more fatal than with two of them, but bad enough...

Needless to say, this was fixed. HDToolBox no longer attempts to (wrongly) set the block size at file system level, and now the block size stays where it should be if you remove a file system from the list.

Offline Aegis

Re: The Os 3.1.4 Thread
« Reply #497 on: July 09, 2019, 10:03:57 PM »
The Click to Front commodity is broken in 3.1.4/1 (because of the intuition-v45 changes?) - please fix for the next release!
Catapultem habeo. Nisi pecuniam amnem mihi dabis, ad caput tuum saxum immane mittam.
I have a catapult. Give me all the money, or I will fling an enormous rock at your head.
 

Offline Louis Dias

  • Hero Member
  • *****
  • Join Date: Mar 2002
  • Posts: 1166
  • Country: us
  • Total likes: 2
  • Gender: Male
    • http://www.diasintegrated.com
Re: The Os 3.1.4 Thread
« Reply #498 on: July 10, 2019, 05:21:13 AM »
The Click to Front commodity is broken in 3.1.4/1 (because of the intuition-v45 changes?) - please fix for the next release!
Do you mean 3.1.4.1.5?
 

Offline Thomas Richter

Re: The Os 3.1.4 Thread
« Reply #499 on: July 10, 2019, 05:41:49 AM »
The Click to Front commodity is broken in 3.1.4/1 (because of the intuition-v45 changes?) - please fix for the next release!
No, it's not. Please read the FAQ. The default tooltypes of ClickToFront are that you need to hold down ALT (or some other qualifier) to make it work. It has been like this forever. Reading the FAQ should definitely be the first step.
 

Offline kolla

Re: The Os 3.1.4 Thread
« Reply #500 on: July 10, 2019, 07:19:55 AM »
Under what conditions does the new SetPatch replace shell with L:Shell-Seg and under what conditions does it not replace shell with L:Shell-Seg? Does it care about what version of shell that is already resident?
B5D6A1D019D5D45BCC56F4782AC220D8B3E2A6CC
 

Offline kamelito

Re: The Os 3.1.4 Thread
« Reply #501 on: July 10, 2019, 07:26:45 AM »
@Thomas Richter
Is a new SDK planned?
Will we see one day updated RKM? those from CBM stopped at version 2 of the OS as you know.
 

Offline Thomas Richter

Re: The Os 3.1.4 Thread
« Reply #502 on: July 10, 2019, 07:44:00 AM »
Under what conditions does the new SetPatch replace shell with L:Shell-Seg and under what conditions does it not replace shell with L:Shell-Seg? Does it care about what version of shell that is already resident?
It's a version/revision match with the Os 3.1.4. shell. If the new shell is already loaded with LoadModule, it won't do anything.
 

Offline Thomas Richter

Re: The Os 3.1.4 Thread
« Reply #503 on: July 10, 2019, 07:49:17 AM »
Is a new SDK planned?
We should definitely make one, yes. It's a matter of priorities, though. Not much was new for 3.1.4, but there will be a couple of news for 3.2 so I'm updating the includes and autodocs as I go. But it still means that all of this needs to be packed and brought into a usable state. Then, one of the known defects of the current SDK is that they aren't "const correct", which causes quite some annoyances with even the SAS/C, and that should be fixed as well.

So, it's a long road. I don't have a milestone for that yet, but I believe it should work like first publishing 3.2 - with no definite deadline at this point - and then continue with the SDK for it.

There is already a new "Shell.guide" that explains the extended synax, the new commands and provides some insight into the new features, but that is already covering the V47 shell (currently in alpha stage), so I'm not going to say much about it at this point.

As for the RKRMs: Oh well, I wish I had all the time.

Offline Thomas Richter

Re: The Os 3.1.4 Thread
« Reply #504 on: July 10, 2019, 05:21:55 PM »
Maybe time for another update on the update. The shell was mentioned already above, so let's continue with this module.

The most prominent bug in the shell was that I forgot to forward the error code from program execution to the topmost level of the shell, which was caused by the integration of pipes into the shell. Kolla noted this probably first. Some people reported problems in the FailAt command, but this was actually fine all the way and the problem was in the shell.

Error forwarding was broken also in another mechanism, namely variable and backtick expansion. If that failed, no error was reported, though it probably should.

What was a bug left over from V45 was that redirection to NIL: did work well with console redirection, thus you could get a spurious "please insert volume console:". Probably nobody noticed the problem in V45 because it was part of the "&" (run back) operator, but now that the same mechanism is also used to run the left-hand side of a pipe, the error became apparent.

Then, we had a bug in the proper handling of $?, which evaluates whether a variable is present. If so, $?a returns 1, otherwise 0. This worked only for environment variables, but not for shell local variables. I believe this might have also been an old bug that remained unnoticed.

Last, I recall that we also had a parsing bug with two variables sitting close to each other, such as $a$a. Since Shell and Execute share the same parser - a useless issue of code replication that will go away next time - the same bug was also in execute, and thus you also have a new version of execute as well.

Speaking of execute, it can now also run at the right-hand side of a pipe, a feature that was somehow "in focus", but I simply forgot to put it in. The best application for this is probably together with "list lformat". You can create a shell script dynamically by "list", and pipe it directly into execute.

There was an APIPE: handler in Aminet that would do similar by executing everthing that was written into it, now the shell can do it natively.

Offline kolla

Re: The Os 3.1.4 Thread
« Reply #505 on: July 10, 2019, 06:40:05 PM »
I found $? so unreliable that I simply dropped using it, and instead use the old 'if "$var" eq "${var}"'-trick, really wishing that non-existing variables could just expand to an empty string.

A shame that all these new features cannot really be used in distributed scripts anyways, since only a fraction of the systems out there may have the shell required.
B5D6A1D019D5D45BCC56F4782AC220D8B3E2A6CC
 

Offline my_pc_is_amiga

Re: The Os 3.1.4 Thread
« Reply #506 on: July 11, 2019, 05:33:48 AM »
Thanks a lot for the update!   This issue with Multiview does not look like got addressed in 3.1.4.1:

https://forum.amiga.org/index.php?topic=73908.0


 

Offline my_pc_is_amiga

Re: The Os 3.1.4 Thread
« Reply #507 on: July 11, 2019, 05:39:23 AM »
Cross-posting a copy command memory leak issue from here:

http://forum.hyperion-entertainment.com/viewtopic.php?f=15&t=4321
 

Offline Thomas Richter

Re: The Os 3.1.4 Thread
« Reply #508 on: July 11, 2019, 06:48:24 AM »
Cross-posting a copy command memory leak issue from here:

http://forum.hyperion-entertainment.com/viewtopic.php?f=15&t=4321
I afraid it is too late now.
« Last Edit: July 11, 2019, 07:12:31 AM by Thomas Richter »
 

Offline Thomas Richter

Re: The Os 3.1.4 Thread
« Reply #509 on: July 11, 2019, 06:51:30 PM »
More on the update.

There is a new version of the FastFileSystem in the update as well. Actually, this is only a very minor update as there was nothing to fix, just a couple of things to improve. So, your data is not in danger.

If the FFS initialized a volume for long file names (DOS/6 or DOS/7), it also wrote administration information for directory caches. This data was actually never used, as the dircache remained disabled, so it did nothing bad. As the data is not needed, the new FFS no longer writes this useless data.

Then, when mounting multiple partitions, the 3.1.4. FFS mounted all partitions in parallel. This might have caused a lot of head movements on mechanical disks, so 3.1.4.1 reverts this to serial mounts.

There is a tiny improvement for users of the omniscsi.device, and probably some other devices in combination of removable media. In case you mounted such devices on a drive with a medium inserted, nothing would happen. The corresponding device does not return a proper result for TD_GETCHANGESTATE, such that the FFS would assume that no medium is inserted. The new FFS performs now a "dummy read" to "wake up" the affected device - this will avoid this problem.

The same issue also existed in CrossDos and the CDFileSystem where it was fixed (or rather, worked around) as well.

The FFS "bitmap" records which blocks of a volume are allocated and which are free. For large volumes, there can be many bitmap blocks, and these are recorded in "bitmap list blocks". Previous versions of FFS wrote the bitmap list blocks and the actual bitmap blocks "interleaved", which makes them slow to read. The new FFS writes them in a contiguous list of blocks. This would allow the FFS in a future release to read them in one single access, improving the startup time. The code for that is not yet there, but there is some preparation for it.

Last but not least, the "disk validator" in the FFS was also improved.  The previous version would read blocks in a rather simple-minded fashion: Allocate a block buffer, read the block, validate the block, release the block, then continue. Thus, a lot of rather useless memory allocation operation were performed, one after another - this made disk validation unnecessarily slow. The new FFS will allocate the block first, then use throughout the validation process, and release it when its done.

Oh, and one final change: The FFS creates (since 3.1.4) its own entries in the "FileSystemResource". In order to avoid problems with "HDToolBox" type programs that do not inidicate a proper stack size (due to a mix up of the unit), the FFS now creates the proper "PatchFlags" to indicate a stack size of 2048.

The next (major) release will include just another mechanism for file handlers to indicate their stack size that is a bit more "fool proof" than the current way... Live and learn....