reading bitrotted amiga floppies

[kamiga]

I thought you guys might be interested to know about some research I've been doing lately.

As some of you may know, I've made an external USB-attached amiga floppy drive controller, that can create .ADF files from amiga floppies using regular PC floppy drives.

There's some detailed links, etc, in this forum thread.
Amiga Floppy Project Status thread

There are a bunch of problems with reading 20 year old floppies and the problems generally fall under the problem of "bitrot."  Now amiga disks can go bad a bunch of different ways, which is also something I'm looking into.

In any event, I have created a method of error correction that just doesn't DETECT where errors occur, it actually fixes them.  When an Amiga writes data to a disk, it encodes that data using MFM.  MFM has a very particular structure --- the embedded clock bits are computed/generated from the data bits.  If you are missing a bit here or there, you can actually reconstruct that bit if the surrounding bits are still intact.

Another interesting thing about Amiga MFM is that because of the structure, there are only certain raw MFM byte possibilities.  There are actually only 32 bytes (out of the possible 256) that are legal raw MFM bytes.  As a result, I can actually tell WHICH bytes in a particular sector are bad.  Also, just like the english language where not every two-letter combination is valid (do you ever see ZX together, or QM together?) MFM has illegal and legal digraphs where not all combinations are valid.  A full 1/3 of them are illegal.  I can also use this to determine where exactly an error is occurring.

Last, but not least, I use a best effort guess protocol where I'm not just brute-forcing an answer to correct the byte.  I start off assuming there are 1-bit errors, then move to 2-bit errors, and then to 3-bit errors. This is per byte.  And remember, the search space is much smaller than you think!

The net result, if you don't care about all the technical details, is that THEORETICALLY, I can recover something on the order of tens of bytes per sector with each of those bytes having up to (3) bit errors each.  I haven't done the math, but the good news is that if you throw more CPU power at the problem, you can recover more data.  There's a point of no-return and I haven't done enough research to know how valuable this is.

Anyways, I thought this was a neat idea, and I've never seen these concepts presented before.

I have very rough initial code done and while it works high level, I've got to integrate the error correction into the existing client software that I wrote that reads the disks.  

Thanks for reading.

Keith

[Colani1200]

That sounds pretty cool! Will the code be for your floppy controller only or will there also be some kind of Amiga program? I tried to recover some stuff I made many years ago from bitrotten disks lately, but Disksalv couldn't completely restore them. So I (and probably many others) would be very interested in a solution that works better.

[kamiga]

Colani1200 wrote:
[W]ill there also be some kind of Amiga program? I tried to recover some stuff I made many years ago from bitrotten disks lately, but Disksalv couldn't completely restore them. So I (and probably many others) would be very interested in a solution that works better.

The software, which is written in Java, is only currently designed to run on the Windows platform, though OSX and Linux is somewhere down the road.  The software has to talk to the hardware via USB to be useful, and so having just software means you have 1/2 of a working solution.  So no miggy software.

Part of the idea behind my project was exactly the point that you don't NEED an amiga or an amiga floppy drive to read amiga disks.....

This really isn't designed as amiga disk repair software(like disksalv etc), as much as it is to help create accurate .ADF's from physical amiga disks.

Now, there's no reason why amiga disk repair software couldn't also use these techniques......

Thanks!

Keith

[Colani1200]

kamiga wrote:

Now, there's no reason why amiga disk repair software couldn't also use these techniques......

That's why I was hoping that you would write one.  :-D

[cecilia]

wow, i may actually be able to read my amiga disks someday??

[kamiga]

Colani1200 wrote:

That's why I was hoping that you would write one.  :-D

Heh. I actually wouldn't mind getting back to writing amiga programs again.  I wrote some stuff in C back in the day.  Using.... I think Lattice.... or the other one, SAS/C.  I think I tried DICE but I could never get it work properly....

[Amithony]

This is a good idea. Why not complete the code send it up to a shareware site? I think many Amigan's will be happy to give it a try. It will be interesting to see what the feedback on the whole would be re: recovery.

My 2c adjusted for inflation.

[kamiga]

Amithony wrote:
This is a good idea. Why not complete the code send it up to a shareware site?  

Complete the code?  You mean start the code.  The error correction stuff will definitely make it into the Amiga Floppy Project on the PC side of things.

http://www.techtravels.org/amiga/amigablog

But now that I think about it, I might be limited on what I can do at 7mhz. Or 25/30mhz. or whatever.

I'll have to think about this and do some math to see if the idea flies on a slower processor.

Keith

[pyrre]

Would it be possible to connect this to a PC running UAE and do the same job?

[Amithony]

pyrre wrote:
Would it be possible to connect this to a PC running UAE and do the same job?

On a machine like yours I think that wont be a problem. How do you find the PPC?

[Eclipse]

Cool idea. I'd make one of these when you are finished.
The thing is, this way I believe is better than repairing a disk as you can always write the adf back to a new floppy.

[Trev]

Have you looked into Arjuna http://www.siliconsonic.de/arjuna/? It could be used to provide the framework for your decoding/error-correcting module. It currently works with Catweasel cards, but it's open source. Perhaps you could expand it to support your USB drive as well? You'd have to be OK with releasing your work under the GPL, though (or contact the author for a separate license).

[kamiga]

Trev wrote:
Have you looked into Arjuna http://www.siliconsonic.de/arjuna/? It could be used to provide the framework for your decoding/error-correcting module. It currently works with Catweasel cards, but it's open source. Perhaps you could expand it to support your USB drive as well? You'd have to be OK with releasing your work under the GPL, though (or contact the author for a separate license).

Nope, I've never seen or heard of that before, thanks for the link.  It appears, and I didn't dig far, that arjuna is designed specifically to read raw data from a Catweasel card. Then the Arjuna modules interpret that data.  My solution is an alternative to the Catweasel --- interfacing via USB instead of a PC card.

As far as licensing goes, I haven't looked much into it --- there are so many different schemes with pluses and minuses.  I intend for full open source of my stuff..... most of it is already online -- although not in any type of organized manner, yet.

Thanks

Keith

[hardlink]

kamiga wrote:

But now that I think about it, I might be limited on what I can do at 7mhz. Or 25/30mhz. or whatever.

Um, why? Are you doing real-time data correction? If not, the only limitation should be time. I wouldn't mind my A500 running 24/7 for a week/month/year to restore a desired disk. It's just sitting there anyway!    :-P

[Trev]

@kamiga

That's the point, though. It could be slightly redesigned to use your decoding mechanism (or preferrably, a toggle for either) and a standard set of interpreters. Or something like that. Plus, the author obviously knows a thing or two about floppy disk hardware. It's a good collaboration opportunity.