Hyperion announces OS 3.1 update

[olsen]

FANTASTIC!  Indeed, thank you, Olaf.  You know we all want to hear about it.

Bit of a dry subject, I suppose... Well, somebody asked for it

The new Disk Doctor is the result of some hairy research into how the original version worked and why it failed. Commodore stopped shipping it with Workbench 2.1, and since then no file system repair or data recovery tool has been included with subsequent Workbench versions for Amigas with 68000 family CPUs.

The new Disk Doctor should solve a couple of hairy problems. Volumes can be much larger than they used to be in the 1980'ies and 1990'ies. Not only do you have to deal with storage media much larger than 4 Gigabytes, the number of files, directories and the associated management data structures have to be analyzed and their contents need to be kept in memory. Memory constraints are the biggest problem here, actually. The original Disk Doctor needed about 1.5% the size of the volume as working memory (RAM). For example, in order to "repair" a 20 Megabyte hard disk partition, you would have to have at least 330 Kilobytes of free RAM available. This would not fly on the original Amiga 500/1000/2000. Now imagine how the math would work out for a 1 Gigabyte hard disk partition. For the new Disk Doctor I developed a special type of data structure which lowers the memory requirements to around 0.1% of the volume size. Which means that about 1 Megabyte may be sufficient to deal with a 1 Gigabyte partition, and 8 Megabytes for an 8 Gigabyte partition. At least, this is what testing revealed so far.

Unlike the original Disk Doctor, the new version does not currently modify the contents of the volume. It only does two things: 1) examine the contents of the volume, looking for defects and 2) copy the contents of the directory tree to a different volume. It does what Dave Haynie's DiskSalv program did 32 years ago, but of course it does a lot more than that

The "examination" begins by looking into every single block of the volume, taking note of the contents, the type of data found, damage to the contents. This is followed by another pass to figure out what files, directories, hard links and soft links exist and can be reached through the root directory (if there is one). Finally, this information is tied together so that one can tell which files, directories, etc. are still "sound" and undamaged, which ones are deleted, damaged or "orphaned", i.e. have no valid parent directories.

This information can be stored in a "disk and block information" (DABI) database file (actually, it's just a gzip-compressed plain text file) which might just become useful later. Instead of rerunning the examination (which takes quite a while to complete on a large volume), you can reuse the information gathered later.

You can use the new Disk Doctor just to check if a volume is in good shape, and not bother with the DABI files or the copying operation. But there's a lot more you can do. For example, you could use the DABI file to have Disk Doctor show you what it would have copied and then narrow down the set of files to copy through very simple filter restrictions (e.g. copy only "sound" files, copy only damaged files, copy only deleted files, copy only files matching a wildcard pattern). The copy process itself works very much like a "copy all clone #? .." command would, preserving all the properties (comment, user/group ID, modification time) of the original directory entries. When damaged files are copied, the damaged sections are skipped. The copy will retain the entire directory tree structure, if possible.

The new Disk Doctor takes a very thorough look at the state of the volume and its data structures. This includes, for example, making sure that all directories are consistent. It's possible for directory entries to show up when you list the directory contents, yet remain inaccessible when you try to open or delete them. The linkage information underlying hard links to directories and files is validated, too. Cycles in the many list data structures which the file system uses are detected. The root directory and its associated data structures (e.g. the bookkeeping information on what blocks are still available for use) are examined, too, which covers the bitmap extension block information. The extension blocks were added in 1987 with the introduction of the FFS. Their "Achilles heel" is the lack of a checksum which would make detection of corruption easier. The directory cache lists which give the directory cache (DCFS) file system variant its name are validated, too, and any differences found between a directory and the cache contents are recorded.

All this information is in part intended for a repair operation which is currently not implemented. A repair strategy would be needed first, and I have yet to come up with one. The more I learned about what can go wrong the more alternatives to dealing with the defects revealed themselves. You may be able to correct the smaller problems, such as restoring the consistency of directories, but if the root directory is damaged, how do you rebuild its bookkeeping information without destroying other data that might still be recoverable? So, after four months of work, this is still a research project, I'm afraid.

Finally, in case you wondered, the new Disk Doctor supports all Amiga file system variants which stem from the 1986 ROM file system and the 1987 Fast file system. This includes the international mode (introduced with Workbench 2.1), the directory cache mode (introduced with Kickstart 3.0) and the long name mode (introduced with AmigaOS 4). Hard links and soft links are supported (currently only soft links are restored by the copy operation, restoring the hard links still needs work). Volume block sizes of 512..65535 bytes per block are supported, too.

I can't promise you that using the new Disk Doctor will be an enjoyable experience (ha!), but it should be a whole lot less exciting than it used to be with the old Disk Doctor. If you need a tool to check if your volumes are sound and in good shape, and which will help you to recover data from them when you really need it, the new Disk Doctor should get the job done.

[kolla]

I suppose everyone knows the "amiga connection" of gdb?

[psxphill]

Due to a matter of bad luck and just coincidence, MungWall now believed that the icon.library allocation code comes from layers, though classified the icon.library (actually exec) memory release call as regular. This is because icon as well as layers are "loadmodule'd" and for that reason, the resident list, which is checked by MungWall, is a bit "unusal", so the mungwall heuristics of detecting "this PC belongs to layers" failed.

So can you fix the resident list to make mungwall happy?

All this information is in part intended for a repair operation which is currently not implemented. A repair strategy would be needed first, and I have yet to come up with one. The more I learned about what can go wrong the more alternatives to dealing with the defects revealed themselves. You may be able to correct the smaller problems, such as restoring the consistency of directories, but if the root directory is damaged, how do you rebuild its bookkeeping information without destroying other data that might still be recoverable? So, after four months of work, this is still a research project, I'm afraid.

When I've done similar data recovery, I've ended up using a "what if" algorithm that tries multiple things when there are ambiguity & picks the most consistent one.
It's a load of planning and a load of code. You shouldn't aim for perfect recovery in all cases though because it's impossible. Getting consistency back means you don't have to choose between reformatting and losing more data when the corruption upsets the file system.

[guest11527]

So can you fix the resident list to make mungwall happy?

There is nothing wrong with the resident list. There is quite a bit wrong with MungWall. Apparently, its heuristics to identify layers is broken, and its missing a version check on it, too, so just don't use it. There are two replacements available for it.

[olsen]

When I've done similar data recovery, I've ended up using a "what if" algorithm that tries multiple things when there are ambiguity & picks the most consistent one.
It's a load of planning and a load of code. You shouldn't aim for perfect recovery in all cases though because it's impossible. Getting consistency back means you don't have to choose between reformatting and losing more data when the corruption upsets the file system.

Thank you, this is good advice. I do tend to spend a lot of time researching a problem and eventually overengineering a solution (why not? I don't like to return to the same project over and over again to fix issues I could have caught at the research & design stage). The simplest solution that works might just be the ticket here.

Hard to believe, but it appears that the original Disk Doctor's purpose was to only get the file system into a state which allows its contents to be copied to a different medium, using the tools available at the time (the "Copy" command, for example). It did just enough to make the basic file system structures work again, even rebuilding the consistency of directories. Broken files, etc. remained broken, sometimes becoming even more smashed if you reran Disk Doctor again. It was all too tempting to view this process as a repair operation, but it was never even that. If you only had one single disk drive, though, you had to hope for the volume to be "repaired", because you could not easily copy its contents to a separate volume (i.e. switch disks for each single file copied).

From that perspective the new Disk Doctor can already deliver what the original command tried to make possible: copy/salvage data from a compromised, if not defective medium.

If the new Disk Doctor is to repair a volume, it should be able to deliver a consistently readable and writable file system. The simplest working approach to make this happen would be to remove all the damaged contents and to rebuild both the root directory and its bookkeeping data structures. I reckon that this is doable without overengineering the solution too much

Because the new Disc Doctor already supports a "preview" feature, one could show which files and directories would be cut. The user could then decide which data to copy before letting the new Disc Doctor make changes which would then result in loss of data.

[klx300r]

..I can't promise you that using the new Disk Doctor will be an enjoyable experience (ha!), but it should be a whole lot less exciting than it used to be with the old Disk Doctor. If you need a tool to check if your volumes are sound and in good shape, and which will help you to recover data from them when you really need it, the new Disk Doctor should get the job done.

thanks & yes much needed as the old DD unfortunately was like playing the lottery (at least for me)...thank goodness for DiskSalv more recently that saved quite a few hairy situations & reminded me yet again to backup, backup, backup:p

[Bennymee]

two months after the last repsonse here, is there any news ?

[Lord Aga]

I would say this is cancelled.

The coder now has a full-time job trash-talking Apollo/Vampire stuff, so there's no time left to actually produce something himself I'm afraid

[Gulliver]

I would say this is cancelled.

The coder now has a full-time job trash-talking Apollo/Vampire stuff, so there's no time left to actually produce something himself I'm afraid

Well, then I am glad, because now the Apollo guys will be forced to implement a proper MMU tu run Linux. Or they could stil resell AmigaOS 3.9 and Fusion mac emulation.

[guest11527]

I would say this is cancelled.

Nope. Where is all this nonsense coming from.

[kolla]

Nope. Where is all this nonsense coming from.

IRC Freenode, #apollo-team - ever been there? It's a crazy place, they make up truths as they go, revision history and badmouth everyone else. It's like a late night dive bar, except they're not even drunk.

[OlafS3]

IRC Freenode, #apollo-team - ever been there? It's a crazy place, they make up truths as they go, revision history and badmouth everyone else. It's like a late night dive bar, except they're not even drunk.

and for what purpose are you there?

[kolla]

and for what purpose are you there?

I am not, I left long ago.

[guest11527]

two months after the last repsonse here, is there any news ?

So, what's new? Been busy the last weeks with a replacement of the CD file system (CDFS). One may wonder why this is actually needed, there are really sufficient replacements out there. The main reason (besides "it was fun to do") was "consistency". 3.1.4 comes with three file systems, and a couple of handlers. For file systems, we have FFS, CrossDos and CDFS. For handlers, we have the port-handler, the queue-handler, the aux-handler, and the speak-handler.

What we also have is a new "Mount" command, and a couple of new mount options that are also supported by HDToolBox. As mount options we have "DirectSCSI" which enables SCSI transport rather than trackdisk-style transport. "NoNSD" which disables NSD-probing since probing for NSD may have bad sideeffects on devices that do not support NSD, and "SuperFloppy" which tells filing sytems "go hunt for the disk geometry yourself".

And we have now - thanks Tony - a fourth file system out there in the wild which already supports these options. That's PFS3.

So, what do we have: A FFS that can be told to "talk SCSI" by a mount option or a RDB flag, to be set by HDToolBox. Clearly, I also updated CrossDos to support that. While at it, CrossDos now also supports long file names, FAT32, FAT24 and FAT16.

Now, it was only logical that CDFS should support the same, so some work was required. The 3.1 CDFS was out of the question (rotten), so I back-ported the Os 4 CDFS to Os 3.1.4, and added the new options, and fixed a couple of things. So, the CDFS will also speak SCSI if you tell it so. It now honours "Mask", "Maxtransfer" and "BufMemType", which it failed to do so. The same goes for CrossDos and FFS , of course.

Last but not least, there is one new feature both in CrossDos and CDFS, and that is "multi-threading". If an application triggers a big read or write from/to the harddisk, FFS is not quite as stupid as one may think: It triggers the read asynchroniously, but then is still ready to take new commands, for example the workbench ACTION_DISK_INFO that comes in about every second. Now, in the past, if you had a read going on from a CD, the workbench was "frozen" after a second or so. This is simply because it does not get its command through when looking for drives. The CDFS was busy, waiting for the data on the CD to arrive for the user process that triggered the read, but could not answer the trivial "are the disks all still there" request from the workbench.

This all changed. Unlike Fat95, "CrossDos" is now multithreaded. And unlike all the other CD file systems I'm aware of, "CDFS" is now also multithreaded. It dispatches the read, then goes back processing other commands, not getting in your way.

CDFS is a backport from 4.0, but after code review, I wasn't happy with all the code I found, so it currently "only" delivers ISO, rockridge, joliet and CDDA support - digital audio tracks appear as files in the file system. I have currently mixed-mode CDs, UDF-support, HFS support and HFS-plus support disabled. Code is under review, and depending on how far we get, might or might not be included in the final distribution.

While at it, the port-handler also became pre-emptive. Thus, it can queue up multiple read/write requests at a time without putting everything on hold.

So, that's what I have been busy the last three weeks: Re-designing CDFS, introducing co-routines, re-viewing code, testing code.

As I'm not the only developer, there are also other news. Since Reaction is gone, the ASL and workbench prefs are gone. We now got replacements for them, based on gadtools. Ok, they don't look "as nice" as the reaction parts, but they fit well to the optics of the remaining 3.1.4 "gadtools" based preferences.

We also got a new "UTF to Compugraphic" font mapping. There are multiple compugraphic fonts out there that seem to miss characters. However, the characters are there, just in places where the bullet library does not look for. All that is hidden in a unicode to intellifont ("dumb-o-font") translation table "if.uc" which takes care of that - just didn't do that right...

We got "Ed" fixed up. This is really a pretty screwy program - probably because it came from BCPL and was "translated automatically to C". Or at least, the code looks like that. Good part: Ed can now run in the same shell window it was called from, and can be run over "AUX:", i.e. a serial connection.

So, all the usual: Bug fixes, and lots of work.

Yes, we're looking for more beta testers as well. More testing is certainly needed.

[kolla]

This CDFS has nothing to do with old L:CDFileSystem (aka cdfs.library 40.11), which be gone? Sounds good to me. There have been some queue-handler replacements around too, don't know if you know about them? One is versioned 50.something, so I wonder if maybe it comes from OS4.0 or OS4.1 for classic.

[Oldsmobile_Mike]

We got "Ed" fixed up. This is really a pretty screwy program - probably because it came from BCPL and was "translated automatically to C". Or at least, the code looks like that. Good part: Ed can now run in the same shell window it was called from, and can be run over "AUX:", i.e. a serial connection.

So, all the usual: Bug fixes, and lots of work.

Nice!  Looking forward to seeing which of these programs I can "hack" to work under my existing 3.9 installs.  Maybe someone should write a guide.  

Would be nice if PFS3 was made the default file system, but I can't really see such a "radical change" being adopted by the community as a whole.  :lol: