Amiga history: Why was "Disk doctor" so spectacularly bad at its job? Here is why...

[klx300r]

@ olsen

thanks for posting! love the insiders perspective

[kolla]

Actually it's even weirder - the default ROM file system reads the volume 'last altered date', and if the system time happens to be unset, will change it to that date.

The way the default file system works, it should be sufficient to change a file on the volume. That change will bubble up to the root directory, and it should replace the 'last altered date' unless that change came with a time stamp which preceded what's already recorded in the root directory.

Are you really sure about this?

I ask because that is not at all how I experience it, for me it looks like the time of formatting, and not 'last altered', is used. I typically keep a loopy script update a .timestamp file regularly, and doing that alone does not work.

Of course there is  I wrote it, starting in 2001, for use on AmigaOS 2.x/3.x. This is still the most complex and challenging software I ever wrote. The AmigaOS 4 and MorphOS versions are ports of the original 68000 implementation. I still use the 68000 version to this day on my A3000UX development machine at home.

Right, and of course.

Turned out my problem was not FFS at all, it was really a partition with SFS/02, that didn't show up in OS3.9 - not FFS with long filenames. I had forgotten to add SFS/02 to RDB. This is my A3000 that multiboots between OS3.9, OS4.1 and Morphos 1.4. I am pondering on moving back to FFS as I keep getting random errors from SFS about not being able to read blocks (750GB ATA drive with ACard IDE-UW SCSI). It's sadly not so easy to find a filesystem that one can rely on for all the different "flavours". Also, I hope Amiga filesystems will improve so that they transparently can deal with bad-blocks, bit-flips etc, as all such errors become statistically impossible to avoid on large drives.

[TheMagicM]

It was before my time but DiskSalv does perform admirably for FFS partitions and was programed by 'hardware' Amiga guru Dave Haynie!!!

OMG... DiskSalv saved my butt a few times.  What an excellent piece of software!!

[olsen]

Are you really sure about this?

I ask because that is not at all how I experience it, for me it looks like the time of formatting, and not 'last altered', is used. I typically keep a loopy script update a .timestamp file regularly, and doing that alone does not work.

Not as sure as I should have been. I just looked over the original Amiga file system code and found the reason why a change to the file system contents need not affect the "last altered date" of the root directory.

This has something to do with how the file system keeps track of which blocks are still available for use, and which have already been allocated. That information is stored in what's call the "bitmap" in which a series of blocks account for all the allocated and unused blocks.

Because the bitmap may not be current, the validation process may have to be summoned to rebuild it. The trigger is stored in the root directory block, which is a flag that indicates whether or not the bitmap is valid. That flag is cleared when new blocks are allocated and the bitmap has not been written back to disk yet, and it is set again when whole operation which eventually required the bitmap update has concluded.

This is where the "last altered date" field comes in: it is updated only if the bitmap contents change. This happens when the validator finishes its job, if you add a new directory entry, add data to a file or delete a directory entry. So, in a nutshell, changing the date stamp of a file is insufficient for moving the "last altered date" record ahead.

I have not looked at the V34-V40 FFS implementation (funny that I can make sense of a pageful of BCPL code more quickly than a dozen of 68k assembly pages), though. My best guess is that it reproduces the behaviour of the original Amiga file system.

Right, and of course.

Turned out my problem was not FFS at all, it was really a partition with SFS/02, that didn't show up in OS3.9 - not FFS with long filenames. I had forgotten to add SFS/02 to RDB. This is my A3000 that multiboots between OS3.9, OS4.1 and Morphos 1.4. I am pondering on moving back to FFS as I keep getting random errors from SFS about not being able to read blocks (750GB ATA drive with ACard IDE-UW SCSI). It's sadly not so easy to find a filesystem that one can rely on for all the different "flavours". Also, I hope Amiga filesystems will improve so that they transparently can deal with bad-blocks, bit-flips etc, as all such errors become statistically impossible to avoid on large drives.

You would not want to go back to the all-checksum OFS blocks, I suppose This is a file system for desktop computer systems after all, which has little CPU time and memory to burn. The "boring" solution to the problem of data corruption is still to keep multiple sets of backups

As for the storage devices involved, they ought to report decoding/consistency errors as read errors, as floppy disks used to do. Not sure if this is still an option today, though, when everybody can just throw CPU power and memory at the problem on the file system level.

[guest11527]

Not as sure as I should have been.

As far as the V45 FFS is concerned (and probably everything below), a date change only bubbles up a single directory level, and then stops there. IOWs, if I change a file in directory A, the date of A changes, but not the date of the parent of A.

Hard to say why this design choice was taken, probably to avoid even more "disk grinding" when writing files.

I have not looked at the V34-V40 FFS implementation (funny that I can make sense of a pageful of BCPL code more quickly than a dozen of 68k assembly pages), though. My best guess is that it reproduces the behaviour of the original Amiga file system.

Now, I haven't checked the BCPL "original", but almost every language is easier to read than assembly (with the exception of brain%&$#?@!%&$#?@!%&$#?@!%&$#?@!, maybe), so that's not much of a surprise.

Unfortunately, the assembly FFS lacks a certain programming strictness in the sense that you never know which registers are actually preserved over calls, so the code is needlessly hard to follow. It's one of the typical "premature optimization" issues...

Certainly, the 68K did have a performance issue with complex code, but is there *really* a need to have 5 almost identical functions that differ only in how far in which *register* they take their primary argument?

[kolla]

Code: [Select]

setdate dh0:
It should just work

[mechy]

Are you really sure about this?

I ask because that is not at all how I experience it, for me it looks like the time of formatting, and not 'last altered', is used. I typically keep a loopy script update a .timestamp file regularly, and doing that alone does not work.



Right, and of course.

Turned out my problem was not FFS at all, it was really a partition with SFS/02, that didn't show up in OS3.9 - not FFS with long filenames. I had forgotten to add SFS/02 to RDB. This is my A3000 that multiboots between OS3.9, OS4.1 and Morphos 1.4. I am pondering on moving back to FFS as I keep getting random errors from SFS about not being able to read blocks (750GB ATA drive with ACard IDE-UW SCSI). It's sadly not so easy to find a filesystem that one can rely on for all the different "flavours". Also, I hope Amiga filesystems will improve so that they transparently can deal with bad-blocks, bit-flips etc, as all such errors become statistically impossible to avoid on large drives.

I gave up on SFS/02  had nothing but problems early on. dropped back to sfs 1.279 sfs/0 and never had a single problem.

[mechy]

Very nice write up on disk doctor olsen. thanks!

[olsen]

Code: [Select]

setdate dh0:
It should just work

It should, but it probably won't

SetDate, Protect and Filenote are the three commands which are not supported for the root directory (which does not have file protection bits, nor a comment field).

All three operations run through the same common function. In the original Amiga default file system, as well as the FFS versions 34 through 45, all three are rejected with the error "object is the wrong type".

Apparently, the only way to advance the "last altered date" field of the root directory is in making a change to the file system which has the effect of allocating/releasing storage space. This should do it:

Code: [Select]

Echo > dh0:change_date
Delete dh0:change_date

[kolla]

Apparently, the only way to advance the "last altered date" field of the root directory is in making a change to the file system which has the effect of allocating/releasing storage space.

Allright, I will test this later tonight, and change my loopy script to something like this...

Code: [Select]

set timestamp sys:.timestamp
lab loop
if exists $timestamp
  delete $timestamp quiet force
else
  echo to $timestamp
endif
wait 60
skip back loop

I cannot help laughing at this silly workaround :laughing:

[mark_k]

Apparently, the only way to advance the "last altered date" field of the root directory is in making a change to the file system which has the effect of allocating/releasing storage space.

ACTION_SERIALIZE_DISK sets both creation and last-modified dates in the root block. It's possible that due to it doing TR_GETSYSTIME once for creation and once for last-modified fields, that the last-modified time is one or more ticks later than creation.

ACTION_RENAME_DISK changes the volume last-modified date (at "size-10") only.

What changes the root datestamp at "size-23"?

[olsen]

ACTION_SERIALIZE_DISK sets both creation and last-modified dates in the root block. It's possible that due to it doing TR_GETSYSTIME once for creation and once for last-modified fields, that the last-modified time is one or more ticks later than creation.

ACTION_RENAME_DISK changes the volume last-modified date (at "size-10") only.

What changes the root datestamp at "size-23"?

Any change in the bitmap will do that. The file system rewrites the root directory as a by-product of marking the bitmap as "valid" again after a write access (added data to a file, truncated a file, created a directory entry, deleted a directory entry) has made changes to it. Before the root directory is written back to disk, the "last date altered" field (at offset size-23) is updated, too.

So the root directory actually contains three distinct date stamps: 1) size-7 is the date and time of the disk initialization ("serialization"), 2) size-10 is the date and time of the last volume update (which only advances in time and never goes back) and 3) size-23 is the time and date of the last change made to the root directory block.

[kolla]

Allright, I have done some testing and conclude how this works on my MIST, which has no RTC to speak of, using FFS 45.16 (if I recall correctly).

When I boot without startup-sequence, the date is 11-Oct-09 03:14:16, which I guess was the time I originally installed this fileystems on the Minimig. I agree that it is somewhat funny that the date is 11-10-09 followed by a clock that looks strikingly similar to pi

There is a SYS:.timestamp there, and if I do this:

Code: [Select]

assign T: RAM:
date `list sys:.timestamp lformat "%D %T" dates`
the system time is set according to what the timestamp of that file is (in my case 16-Sep-17 16:40:15).

According to above speculations, it should now simply be a matter of deleting the sys:.timestamp, and the "last date altered" of the filesystem should change. And maybe it does, who knows... I don't know of a operating system tool that can actually read the mentioned timestamps of a filesystem. What I do know, though, is that a reboot sends me back to 11-Oct-09 03:14:16.

So, it appears to me that adding and removing files to the root of the filesystem has no effect on the timestamp of the filesystem that is used to set system time on boot, or that something else goes on. Also, relabelling sys: had no effect on system time after reboot.

[guest11527]

So, it appears to me that adding and removing files to the root of the filesystem has no effect on the timestamp of the filesystem that is used to set system time on boot, or that something else goes on. Also, relabelling sys: had no effect on system time after reboot.

No, it does not. The construction is somewhat strange... it's Amiga, after all. One might guess that it is actually the job of the dos.library to somehow find a useful system time from the rtc and use mounted volumes as a backup. However, that's not how it works...

It is the FFS restart function which does. If it detects that the dos date is unset (that is, if the time in the dos "root node" is all zero), it takes the *creation date* of the disk it currently mounts and places it into the dos root node.

Changing the root directory or the bitmap does not modify the creation date of the volume, of course, and hence it does not change the date the system gets.

Of course this also implies that the date on a non-rtc system depends on the volume you boot from, and this can be of course in the past of another volume in the system. Sick!

What dos should really do is take the max of all volumes it is aware of, and FFS should stop fiddling with the system time, and the date should only be set if the system knows that there is no rtc. That FFS needs to mess with the system clock is only for the consistency of the FFS itself (as in: All files on the disk should be created later than the creation date of the disk, which is sensible)....

Surprisingly, Tripos aka dos.library v33 did something different. Back then, "FFS" was not a separate source, but part of the monolithic junk that came from Tripos, and *there* we find

Code: [Select]

AND bumpdate() BE
$(
   LET v = VEC 2


   datstamp( v )

   // only reset the date if this is the boot disk and time is unset
   IF unit.no = 0 & v!0 = 0 THEN
   $(
      IF comparedate( v, date.scan ) < 0 THEN
      $(
         setdosdate( date.scan )
      $)
   $)
$)
IOWs, the date is only set on unit zero (hardcoded!) and the days of the current date are zero (remember the BPCL indirection operator "!"), and then a comparison of the date with the already set date is made.

This looks equally wierd, but at least there is a test not to set the date backwards.

[kolla]

Just to spin on this a little more, though a bit off-topic - how I keep system time without battery backed RTC - I use this wonderful little gem: http://aminet.net/package/util/time/TimeKeeper and I have many times wished that it could be built into a rom module, so that system time could be kept through reboots without the need of running a program from startup-sequence. Just an idea