Hyperion announces OS 3.1 update

[olsen]

So does this mean we will now have a "fork" of AmigaOS 68k at 3.1, so we'll have 3.9 which has to fall the way-side for official updates while 3.1 gets official updates?

If you put it like that, it is a fork of sorts, because it has to be.

To the best of my knowledge the prerequisites for continuing development on the AmigaOS 3.9 project "evaporated" when Haage & Partner and Amiga International either changed their business model or shut down, respectively. More than 15 years have passed since then, and many of the bits and pieces which made up AmigaOS 3.9 have long ceased to be available for further development work. For example, all the updated preferences editors are gone.

By (hypothetically) "rewinding" AmigaOS 4 you will not produce a product which connects straight up to the AmigaOS 3.9 development path, allowing the end-result to be officially supported with further updates and bug fixes.

What we ended up with in the form of the "AmigaOS 3.1.4" project (note: don't get hung up on the "code name", as it might actually change in the future) is a compromise which attempts to bring all the components together which are part of the AmigaOS 3.9 update that are suitable and supportable. It is something in between AmigaOS 3.1 and AmigaOS 3.9, yet also has features which were never available in either of these.

[olsen]

But why? Is it to allow the exotic features of the Vampire boards to be fully supported by Workbench 3.1.x developers? I fail to see how the ability to develop for 3.1 going forwards is going to be better for the casual end user than just dusting off 3.9. If you want something up to date then you'd look at OS4.x not this crusty old grey and blue relic surely?

There is a need for AmigaOS on the 68k platform to be cared for. As helpful as the changes in the AmigaOS 3.5 and 3.9 updates were, once development ceased it became more difficult to find solutions for bugs and limitations. Patches and reworked operating system components helped to close the gaps, but it should have been easier to achieve these remedies. Source code access, and the ability to build complete and working Kickstart ROM and Workbench disk sets from that source code should deliver that.

Assuming responsibility for solving such problems is what the AmigaOS 3.1.4 update (and the work leading up to it) is all about. While you cannot assume that equivalent functionality to AmigaOS 3.9 could be delivered in the same time frame the AmigaOS 3.5/3.9 updates required, at least we can make a start and listen to user and developer feedback.

Let's see where this is going to take us. It ought to be an improvement over the stagnation of the past 15+ years in the AmigaOS 68k area, as the minimum requirement.

[olsen]

Will this also include all the updates that were part of "3.X"?

Everything that's in the box directly comes from either the original Commodore code base or from material licensed for inclusion in the AmigaOS 3.5/3.9 updates for which Hyperion has a license to use it. All the changes subsequently made to this code are completely home-grown.

There is bound to be functional overlap with the "3.X" code, but as far as I know no code is shared between it and the 3.1.4 update.

[olsen]

@gizmo350

That's a great idea! Hyperion should include an install option to update current OS3.5 or OS3.9 systems in addition to fresh installs. That way it could double as a Vampire V4 install disc or a Classic machine library update disk :cool:

Yes, this would be very worth having. Figuring out what to copy, and what to omit, whether there are potential conflicts between the different versions (e.g. SetPatch, IPrefs, etc.), will be a really tough nut to crack, though...

I can see this happening, eventually, but I expect that the initial release of the 3.1.4 update will focus on getting the update ready to ship.

[olsen]

Thomas and Olaf, I'm very excited about what you are doing.  I very much look forward to trying it out.   Thank you!  

I'm guessing that your improved 3.14 will be compatible with all Amigas, even the most basic configurations?  Asked differently, your improvements and bug-fixes will not crash an otherwise stock A500 that is equipped with an updated rom?

That is very much the plan. The baseline build of the operating system and its components is still all-68000 code, with only the ROMs for the Amiga 1200 and Amiga 4000(T) requiring an 68020 (the A3000 expects an FPU to be present, but that's a different story).

While building operating system components specifically for the 68020 and beyond would result in significant ROM space savings, as well as disk space savings, it would open a big can of worms.

[olsen]

I did not say it IS secure, I'm saying that should be one if possible direction of development.

The security aspects have been discussed before, but the Amiga operating system architecture is not designed with security in mind. If anything, the focus was on making the most of very little available RAM and squeeze as much performance out of the CPU as possible.

Security measures are built from layers of isolation which limit what a program or operating system function may accomplish. These are completely absent in the Amiga operating system. Compare this to the then (1985) contemporary Macintosh or Atari TOS designs which use "software interrupts" to call operating system functions. There is at least a context change implied, which could be leveraged to enforce permissions to perform an action. Nothing comparable exists on the Amiga, except perhaps the Envoy server which has a single point of entry (authentication) to either grant or deny access to a resource.

Retrofitting operating system designs not intended to offer security measures typically produce compromises which end in tears.

You could redesign the system, of course, but then you'll invariably have to "triage" the software that is expected to still work with it. A more secure Amiga operating system is guaranteed not to work with the vast majority of currently available software.

[olsen]

(New filesystems)

Have you pondered anything about how the Amiga filesystems perform with non-seektime storage like SSD/flash? Has some of the old tech choices become better or worse?

As far as I can tell the file system design choices made back in the late 1970'ies have not aged well. The file system does not have an effective strategy to organize the layout of directories, and the data structures associated with files so as to minimize access times. This is a major drawback which SSDs alleviate.

Even with these limitations out of the way, the design is still not sufficiently scalable and is still vulnerable to damage and corruption caused by the lack of journaling.

The file system design was adequate for the technology of the time it was created for, which would have been hard disk drives no larger than 20 Megabytes. It still strikes me as odd that this file system was used for floppy disks which were significantly slower than those hard disks and were a big source of trouble on account of the file system using a write cache.

Any minor tunings that can align better to this?

It definitely helps to use larger block sizes (2048 bytes or more), if the increased size does not negatively impact overall data throughput (changes are that throughput will be higher, though). The larger block sizes have beneficial effects on how file and directory data structures may be used. The tables which these data structure contain are larger, which reduces seek times, etc. Mind you, larger block sizes help with mechanical electromagnetic storage devices, too

That aside, the file system design provides little to no leverage to improve its performance by minor tuning exercises. It is, as I mentioned, a design which has not aged well.

[olsen]

Olaf is currently working on a new DiskDoctor to address this issue.

Um, well, this is still a work in progress and based upon a rewrite from scratch (there is no sense in rewriting or porting the original DiskDoctor). The goal is to make a command line tool which does three things: a) detect and report file system damage, b) recover data from a file system and c) repair any damage found.

The detection operation is a prerequisite to recovery and repair, and it will never modify the contents of the file system. You can save the information which the detection process gathered to a file and then use that file as the basis for recovery or repair later, or you can of course rerun the detection before recovery or repair are performed.

The recovery operation will likely work similar to how DiskSalv would do the job, with the difference that you would have a choice to copy only those files which are sound (like a "copy #? all clone" command), only those files which are not known to be sound (deleted, corrupted, etc.), or to perform a "deep salvage" operation which would copy everything that is even remotely recoverable, including data file fragments.

The repair operation would excise any damage found, returning the file system to operational state again. This is what the original DiskDoctor was intended to deliver, except that it rarely accomplished even that, and could be depended upon to leave the file system in a much more damaged state than it was before. You would use the repair operation only if you had previously recovered all the salvageable data.

The new DiskDoctor handles large volumes, on storage media larger than 4 Gigabytes. It supports all Amiga ROM file system flavours including dircache and long name mode, with block sizes up to 65536 bytes. Because large volumes need to be supported, covering hundred thousands of blocks, the new DiskDoctor is specially optimized to use as little memory as possible for its work. You may still need 10 Megabytes of RAM for very large drives, mind you.

This is a rather large project, I'm afraid and I cannot currently quote a release date.

[olsen]

Isn't Olaf the author of FFS2 written in C? Wouldn't be better base fixed ffs on these sources?

Tough one. The FFS reimplementation which is used both by AmigaOS4 and MorphOS was designed and developed on AmigaOS 3.1. It has worked on 68k since, well, the day it actually started working properly

The FFS reimplementation would be a better foundation for future file system development work than the assembly language version. One of the drawbacks, compared to the assembly language version, is that the FFS reimplementation needs much more ROM space. Because it tries to do better in avoiding data corruption, the FFS reimplementation deliberately flushes modified blocks to disk, slowing write operations.

In the mid/long term perspective, I see the FFS reimplementation supporting journaling using write-ahead logging, which would solve many performance and reliability issues. I have a cunning plan how to make this work which could be compatible even with AmigaOS 2.x (which would not support the journaling, though).

[olsen]

It will be interesting to compare these with PFS/SFS.  Any predictions which will be "better"?

Always bet against FFS. Both SFS and PFS, in spite of limitations and bugs which may still be present, can't help but kick the FFS to the curb without even trying.

[olsen]

If you're adventurous, you can test out Toni's patched FFS 45.13 which fixes some bugs and enables DOS\8 long filenames! See this post on EAB.

When it comes to the FFS, one man's adventure is another man's daredevil stunt with tragically comic consequences.

Heed the warning: make backups of the data and check that you can restore them when needed, and only then maybe give this a spin.

[olsen]

Oh well, it really depends on what you are doing. As far as raw data transfer is concerned, neither file system can do any magic. It just has to take the user blocks, and direct the device to write them out. So far, so simple.

The things which the FFS does well it really does better than any other Amiga file system. For example, the FFS performs asynchronous read operations with very little overhead, and nearly zero CPU load if the underlying storage system supports DMA. There's even a demo created by Commodore which shows off streaming a pre-rendered animation (generated by "VistaPro"), the use of the double-buffering screen API in graphics.library, and rendering animations on top of that.

However, what the FFS does well is a small subset of the operations a file system generally has to get done properly, because they occur so frequently. This is where the FFS does not compare well against other Amiga file systems, except perhaps CrossDOS or CDFS (irony be damned).

What ought to work well is storage management, as in finding free space to store new data in quickly, while at the same time keeping fragmentation at bay and just maybe reduce seek latency at the same time (releasing storage space to the free pool is the flip side of this coin). This is doable, but it will cost you, e.g. XFS does this very well, but it has to throw a lot of memory and storage space at the effort.

Directory scanning, including retrieval of metadata, ought to be fast, too. The FFS has never scored well here, because it does not separate directory lists and file metadata. You should not have to wait for the directory contents to arrive in dribs and drabs. It makes the floppy/hard disk appear to be much slower than it actually is. The original 1.x Workbench tried to sidestep this problem by recording the contents of the directory in a ".info" file, but it only went so far: instead of having to walk through the entire directory list and reading the icon files it found, the ".info" file only helped to cut the scanning process short. Finding and reading the icon files remained dead slow because the file system didn't care about seek times or fragmentation.

I wish there were some convenient remedies in reach to address the shortcomings of the FFS design, but the only solutions I can imagine right now begin with introducing journaling with write-ahead logging. Once that can be made to work, we could enable dircache mode, which would instantly improve the directory access performance. Making storage management work better is particularly difficult because there exist no data structures in the FFS design to assist the process.

In any case, resolving these limitations will have the probably unwelcome side-effect of requiring more RAM and more storage space than the FFS needs today. Could be this will be a worthwhile trade-off, but none of these possible changes will come together without a lot of prior work and testing. We are still kinda short-handed in brushing up AmigaOS 3.1 for more exciting things to come, so all of the FFS changes might have to wait a little while longer.

[olsen]

Hi, I have a couple of questions regarding the 3.1 update:

Will there be a CD32 workbench version support? Or will CD32 users still have to resort to hacks and custom built workbench cdroms? (freeanim+joymouse+setcdreboot+cdaudio+OSK)

Sorry, this has not been discussed yet. Building a CD32 ROM should be possible with a minimum of fuss, using the CD32-specific components last updated in 1994. But so far nobody has even investigated if it would be possible to build them from source.

So, you might want to wait until we've shipped something and just maybe we could have a look at the CD32 bits to see what can be made to build. This is rather exotic code to begin with, mind you...

And then, what about also adding CTDV workbench support with the RMTM command?

The CDTV code is closely linked to the Kickstart 1.3 ROM code. Nobody knows how to build any of this today, except perhaps the original CDTV project members, including Carl Sassenrath, and we haven't asked them yet. So, the short answer for now is "probably not, because we haven't got enough of a clue to make it happen".

I for one would like to see more light shed on the CDTV code, which is frankly pretty amazing stuff which was so far ahead of its time that it was almost tragically destined not to achieve lasting impact on the state of the art.

HDToolbox unfriendly brother prod_prep will be updated to support large drives?

Sir, I bow to you, because you just made my day merely by asking that very question

Yes, "prod_prep" has been updated by yours truly to support large drives, to the general puzzlement of everybody involved in this AmigaOS 3.1 update project (most Amiga users don't know what "prod_prep" does, some are probably afraid to ask, or embarrassed to admit having wrecked their hard disks by accidentally using it). However, no amount of rewriting can remove all the quirky & twisted limitations of that runt of a partitioning program. Right now "prod_prep" is generally safe not to make a big mess/crash/set your cat on fire if it encounters a drive larger than 2 Gigabytes. If my calculations are correct, the upper limit are 2.2 Terabytes, but you would be very "brave" indeed to allow "prod_prep" to set up a partition scheme for you on a disk that large.

We really ought to have a safe and sound baseline RDB library which would support all RDB data structure management operations, including reading and writing them. A robust "prod_prep" and "HDToolBox" could be built from that library, and hard disk controllers which deal with RDB data could rely upon a single tested library for that purpose. Right now we have none of that, with hilarious consequences.

Will you also include the quite handy but never directly included "reboot" and "findresident" commands?

These are part of the "Install3.1" disk "C" commands, if I remember correctly, so in a way these are still included, just not installed by the installation script. I believe you would like to see them installed among the regular shell commands, is that correct?

[olsen]

Is that a different dircache mode to Commodore's one?  IIRC that had some shortcomings and wasn't recommended to be used, although I forget why.

It's the same thing, and just to confuddle and befuse everybody, it has several different names (DCFS, fastdir mode, dircache mode).

The shortcomings are finally documented in the Wikipedia article (https://en.wikipedia.org/wiki/Amiga_Fast_File_System), which links to the low level documentation on the DCFS and LNFS flavours which I wrote so that finally there was something that Wikipedia could quote

[olsen]

Rights extended to them by whom?  A long dead company?  Any agreements they had with a company or companies that are now out of business are null and void and I'm certain they were never extended a perpetual license, so stop spreading this crap.

Haage & Partner licensed or aquired code from third parties which is part of the AmigaOS 3.5/3.9 update. They also created code in house which is part of AmigaOS 3.5/3.9, and they also paid developers to create code for it. The company still owns this software and, to the best of my knowledge, has not licensed or sold it yet.

None of this specific code owned by Haage & Partner is tangled up with the Amiga International-licensed AmigaOS 3.1 code. Even if Haage & Partner wanted to, they could not "bake" a fully legally sanctioned AmigaOS 3.9 out of their properties and AmigaOS 3.1.

Yes, it sounds weird, because it is. This is the Amiga business, after all