Welcome, Guest. Please login or register.

Author Topic: Hyperion announces OS 3.1 update  (Read 31512 times)

Description:

0 Members and 1 Guest are viewing this topic.

Offline olsen

Re: Hyperion announces OS 3.1 update
« Reply #14 on: November 18, 2017, 08:20:28 AM »
Quote from: ferrellsl;833282
Rights extended to them by whom?  A long dead company?  Any agreements they had with a company or companies that are now out of business are null and void and I'm certain they were never extended a perpetual license, so stop spreading this crap.
Haage & Partner licensed or aquired code from third parties which is part of the AmigaOS 3.5/3.9 update. They also created code in house which is part of AmigaOS 3.5/3.9, and they also paid developers to create code for it. The company still owns this software and, to the best of my knowledge, has not licensed or sold it yet.

None of this specific code owned by Haage & Partner is tangled up with the Amiga International-licensed AmigaOS 3.1 code. Even if Haage & Partner wanted to, they could not "bake" a fully legally sanctioned AmigaOS 3.9 out of their properties and AmigaOS 3.1.

Yes, it sounds weird, because it is. This is the Amiga business, after all :(
 

Offline olsen

Re: Hyperion announces OS 3.1 update
« Reply #15 on: November 19, 2017, 08:29:39 AM »
Quote from: Gulliver;833287
@olsen
@Thomas Richter

Thank you both for explaining the difficult situation regarding both the CDTV and the CD32.

I believe prod_prep is often forgotten, underlooked and unapreciated, and it is a very powerful tool that all amiga users should know about. Dealing with it, is certainly not easy, and could be made a little bit less difficult by including an amigaguide manual with plenty usage examples.
"prod_prep" certainly lacks documentation (as in: there is none even in the Commodore archives, unless you count the actual source code as the only documentation). Looks like now we'll just might have to write some after all.

While "prod_prep" is useful, its inner workings are not as robust as they should be. The command processing would need to be rewritten, for example, and we haven't even touched the tricky parts which deal with media size detection that are unchanged since about 1991 (the code assumes a sector size of 512 bytes, which is how the maximum disk size limit of 2.2 Terabytes comes about). The current state just about works, but it's not on the same level as HDToolBox is right now. Question is whether "prod_prep" really needs that extra bit of work & polish to make it look, well, like somebody actually cared about making it work properly and reliably.

From how it looks, "prod_prep" was branched off HDToolBox at some point in the late 1980'ies. While HDToolBox was still being updated until about 1993, none of the changes seem to have found their way back to "prod_prep". This was really an in-house tool, and not meant for anything else. It replaced even older tools (1986/1987) which the engineers put together who made the first Commodore hard disk controller work.

Quote
I personally use prod_prep for deploying my own customized system/workbench to my amigas from within a simple script. It is very small and it is a formidable tool for emergency install floppies, where space is severely restricted. This way you save space for other important components and dont need to put both format and hdtoolbox which are quite large compared to prod_prep. Furthermore, it is a bonus that you are not required to use it in an interactive way and that you can fully automate it.
Yes, "prod_prep" is a unique solution to a whole bag of problems. The scriptable partitioning tool is useful, if not essential. If only the implementation weren't so rickety from top to bottom :(

Quote
What about picture.datatype? Do you still have the v45 sources? And even a v45 rebuild without the infamous StormC compiler would be apreciated. I dont enjoy the fat binary aspect of its last incarnation in 3.9. I dont want to deal with code that I wont use. But I still want the 24 bit support.
And of course, I wont even need to mention all the datatypes that need updates. It seems a lot of work is here waiting to drive you crazy. :D
Yes, the code is still around, minus the PPC bits which came from Haage & Partner.

I'm not so sure we'll have to open the whole datatypes classes collection of 100% natural organic canned worms right now. In my opinion the basic classes (sound, picture, text, AmigaGuide) should work sufficiently well for a little while longer. If I remember correctly, I put the 24 bit ILBM support into picture.datatype (with quantization and dithering) and the ilbm.datatype, so that should still work.

Quote
Regarding audio.device: do you have some plans to somehow incorporate AHI into it, so that old applications can finally become more friendly with soundcards? Afterall, unlike the RTG scenario, AHI is open source and it has become the unique and default standard for modern audio applications with no other counterpart attempting to replicate that on the Amiga.
I believe that audio.device is not supposed to be replaced in ROM. It has a peculiar API (ha!) and offers functionality which is not available in AHI because audio.device is not intended to be more than the thinnest possible layer on top of the sound hardware (it's also possibly the oldest operating system component in ROM, virtually unchanged since Sam Dicker wrote it in the dawn of the Amiga). So thin that in the "good old days" software was making assumptions of how audio.device used the hardware.

Whether AHI can be layered on top of audio.device in the future is a different question. Again: 100% natural organic canned worms waiting for somebody to get careless with a can opener ;)

Quote
Speaking about usefull commands, yes, my idea was to have both reboot and findresident installed. BTW, will you also include "warnifpressed". I believe you have the sources for all of these.
Yes, "warnifpressed" is available, too, but it's not one of the installer dependencies. It needs lowlevel.library to be installed, which is in ROM on the CD32, but not on the AmigaOS 3.1 installation disk.
 

Offline olsen

Re: Hyperion announces OS 3.1 update
« Reply #16 on: November 19, 2017, 09:01:44 AM »
Quote from: kolla;833314
Hope the new "ugly" HDToolBox has meaningful ways of dealing with small partitions on huge drives, like being able to specify partition size in MBs (or MiBs in newspeak).
The HDToolBox GUI is incredibly resistant to change, and we did not succeed where scores of other developers tired themselves out before us (actually, make that three developers in total until development ceased in 1993).

The original HDToolBox GUI for the A590 and A2091 was created using PowerWindows. At the time (1988/1989) it was arguably the best-designed GUI in an Commodore-created Amiga program (seriously: internal Commodore communication mentions this). The limitations of PowerWindows (as in: little attention was paid to how Gadgets work, how fonts and window border sizes affect the placement of user interface elements) made it necessary to rewrite the GUI code for AmigaOS 2.0, which actually took 2-3 years to conclude. The result is unmaintainable, I'm afraid and I really feel for the developer who painstakingly reproduced the entire PowerWindows GUI with gadtools.library: that must have been the proverbial job from hell. This is why the specific changes that are needed are currently not within our reach :(

I mentioned that the scriptable runt brother to HDToolBox ("prod_prep") looks like nobody cared enough about it to make it work well. In HDToolBox the situation is similar, but more disheartening. The lack of care affects both the basic RDB management and the media queries (sector size, number of sectors, etc.) and GUI. Normally, one would throw everything away and start over from scratch, and guess what, this is what was done for AmigaOS 3.5 and AmigaOS 4, but we are unable to use that code. Right now we don't have the manpower to crack this problem.

HDToolBox is in its current form a provisional solution which will have to do until there's more time to make a better one. That could begin by writing a robust library for RDB and media queries, and to build on top of that. What went wrong with HDToolBox and "prod_prep" also affects the mass storage device drivers ("scsi.device") which used to have problems dealing with media that would not use 512 bytes per sector. Although the RDB specs do support these, your typical controller software only attempts to read and parse RDB blocks of 512 bytes.

Mind you, in the 1990'ies it was assumed that sector sizes would be 512 bytes, and strange things happened if this was not the case. The old HDToolBox contains special code to detect CD-ROMs and tried not to crash or hang the SCSI bus because these devices use 2048 bytes per sector, and some of the early CD-ROM drives would actually hang if you tried to read fewer bytes per sector. Fun fact: the Amiga Unix SCSI driver didn't know that its read buffer was too short to read a full CD-ROM sector, but tried anyway and ran smack into a kernel panic. So this was pretty much par for the course back then :(
 

Offline olsen

Re: Hyperion announces OS 3.1 update
« Reply #17 on: November 20, 2017, 07:13:24 AM »
Quote from: ExiE_;833350
Can you please explain this a bit - "but we are unable to use that code" - are you talking about technical problem or is it copyright again. I am wondering if you are able to use bits of OS4 where possible, or is it out of your reach even though it is both Hyperion business?
First few years OS4 was at least partially running on 68K, so there should be lots of C code replacing old asm ones...


The solution that was used in AmigaOS 3.5/3.9 was developed specifically for these products. As far as I know the contract under which it was developed is still in effect and Haage & Partner controls the use of the software. This is why we cannot use it.

The solution that is used in AmigaOS 4 was licensed for inclusion in the product. Hyperion does not own it. We cannot license it for use in AmigaOS 3.1.4, and even if we could, the size of the application and its reliance on a GUI framework make it somewhat too unwieldy for inclusion. Warts and everything, HDToolBox is pretty small for what it does, and it fits on the installation disk without a problem.
 

Offline olsen

Re: Hyperion announces OS 3.1 update
« Reply #18 on: November 20, 2017, 12:22:11 PM »
Quote from: Gulliver;833362
@Thomas Richter
@olsen

Thank you for being so open and honest to discuss about the update development. Your answers are truly interesting.
Questions answered by software engineers can have interesting consequences (as in: reduced number of sales, lawsuits, etc.). I'm keeping my fingers crossed that we won't say anything we'll have to regret later, or regret more than we can already expect to regret given the state of the code and the manpower required to update and test it...

Quote
Will it feature a less broken cdfilesystem (like cachecdfs in 3.9) and will it have UDF support this time?
I believe that CacheCDFS was licensed for inclusion in AmigaOS 3.9. We currently try to do with what we have (CDFS, originally created for the CDTV by Carl Sassenrath), and for which Hyperion already has a license.

Quote
And please, dont waste your time updating for large drive support for the also ancient HDBackup + BRU twin brothers. I have never seen anyone use that pair, and probably for good reason.
BRU was originally intended to be used for data exchange between Amiga Unix and AmigaOS. This actually worked, if you used the A3070 tape drive. It also worked with floppy disks, using the same logic that must have been used for multi-reel tape backups back in the days of Unix version 7 ;)  This shows both the flexibility of the software, as well as its adaptability.

The Amiga version of BRU was a strange beast, as it worked almost exactly like the Unix version, with platform-specific adaptations (e.g. hard and soft link support). The HDBackup front-end was added so that you would not need to use BRU in the shell.

BRU goes back a long time and was either created or co-created by Fred Fish (I don't remember the details, but there's your Amiga-connection). The company which created the Amiga port and HDBackup (Software Enhancement Technologies) used to be around until recently. BRU is still a commercially supported product, just not on the Amiga. Chances are that your BRU backups from way back in 1991 are still readable with the modern BRU.

Anyway, the BRU and HDBackup code are not really useful anymore (unless you need to dump those old BRU backup tapes before they decay) and upgrading either to "modern" standards is not planned. Back in 1990/1991 they may have been state of the art, and BRU was possibly even "grossly over-engineered" for use on the Amiga. Today we have alternatives which are better adapted to the Amiga platform.
 

Offline olsen

Re: Hyperion announces OS 3.1 update
« Reply #19 on: November 21, 2017, 11:42:55 AM »
Quote from: chris;833405
That'll screw up all dates that are stored as a 32-bit integer.  I suspect this includes FFS.

The date stamp information used by the FFS, all file systems, dos.library, etc. is genuinely and thoroughly "pre-screwed" already (thank you very much, Tripos). It is good for at least another 5 million years, although I suspect that the time and date conversion inside dos.library will have rolled over several times before the signed 32 bit day counter reaches the ends of its tether ;)

Modern man has been around for only 1 about million years, so I guess we could pin our hopes on other intelligent species to arise and maybe find a fix for that pesky year 5,883,516 problem in what the ancients used to call the one true operating system.
« Last Edit: November 21, 2017, 11:53:10 AM by olsen »
 

Offline olsen

Re: Hyperion announces OS 3.1 update
« Reply #20 on: November 21, 2017, 11:52:22 AM »
Quote from: fx;833403
Backticks exists in unix-shells aswell but the gain is that you can nest things with the $(command) syntax. Now you sometimes need to break things up on multiple lines to obtain the same result. I don't think the exact same bash-like syntax would be necessary but I would really like the ability to nest commands if possible!

Something like:
Echo "$(dir $(cd))"
Which is a very bad and useless example but I just tried to come up with something quickly :)


The basic problem we have with the shell is that it's not designed to be extensible. You cannot currently write a shell script which is able to probe for extended shell capabilities. Mind you, the shell is not an interpreter, and it does not have a consistent language as such. Sure, it's Turing-complete, but that doesn't mean it's useful (your basic Turing machine is Turing-complete, too, but it doesn't do anything beyond that).

If I remember correctly, the complete shell "language", such as the portion which covers variable expansion, escape characters, brackets and other arcana, is still underdocumented to the point that you could drop the letters "der" from that word and still make sense. Let's not put lipstick on that pig just yet ;)
 

Offline olsen

Re: Hyperion announces OS 3.1 update
« Reply #21 on: November 26, 2017, 09:35:54 AM »
Quote from: kamelito;833569
Good. So I suppose that there's a plan to publish all developers information to date for the OS. Something like wiki.amigaos.net but for AmigaOS 3.1.x and maintain it if the developer API evolve. To my knowledge only 2.0 is covered by the RKRM.

Kamelito


This is definitely in the cards. For AmigaOS4 the complete 3rd RKM text was imported into MediaWiki (from the original manuscripts), revised and subsequently updated to cover AmigaOS4. The same could be done for AmigaOS 3.

However, getting everything revised and updated for AmigaOS4, including the illustrations, was a huge effort that took its time. So, please be patient...
 

Offline olsen

Re: Hyperion announces OS 3.1 update
« Reply #22 on: November 26, 2017, 10:36:58 AM »
Quote from: kamelito;833571
Sure but, when it was imported in the wiki all the docs where the 3rd RKRM with nothing specific to OS4, so it has already been done. Why not go back to the point of the first import? Except course if you do not share with them. Kamelito


Yes, that's the point I was making. However, importing the text is but just the first step of making the contents available.

For example, the illustrations need to be imported as well, which is a tad difficult. These are PostScript files, and a good number of these are missing and cannot be regenerated from the original Professional Page files due to corruption. For AmigaOS4 that problem was solved by redrawing the missing illustrations. The illustrations were then rendered to PNG image files.

If I remember correctly, the import process generated artefacts in the wiki text which had to be found and repaired manually, and then there's the issue with hyperlinks.

All of that was done for the AmigaOS4 version, and it's a time-consuming process which, while doable, requires a lot of effort.
 

Offline olsen

Re: Hyperion announces OS 3.1 update
« Reply #23 on: December 29, 2017, 12:56:37 PM »
Quote from: kolla;834466
How bad does it have to be, before you throw your hands in the air and decide to rewrite from scratch? :)


I sort of dragged Thomas into this mess, having taken on the task of fixing the last two big issues with the 2016 AmigaOS 3.1 Workbench disk set release: the hard disk partitioning software. Thomas then thoroughly reworked my rewrite, and the prodprep command is part of the package. HDToolBox had its problems (still has some, owing to how the user interface was integrated with the nuts & bolts partitioning code), but prodprep's problems had their own problems, and then some. Both HDToolBox and prodprep are based upon common code which diverged over the years and each developed different issues of their own (e.g. prodprep leaked memory like you wouldn't want to know, HDToolBox got this much better under control).

Rewriting prodprep from scratch is something that is bound to be on the agenda some day. The state of the hard disk partitioning software that ships on Workbench is such that rewrites and fixes can only yield ever diminishing returns. The nuts & bolts partitioning code just about squeaks by in getting its job done. It fails to implement the full RDB spec, it does not validate existing data structures, etc. This bit has to be redone. Once it has been redone, a saner prodprep and HDToolBox pair could be built on top of it.
 

Offline olsen

Re: Hyperion announces OS 3.1 update
« Reply #24 on: December 30, 2017, 08:37:27 AM »
Quote from: Gulliver;834483
It seems that the entire HDD management code is hanging on some nasty code leftovers and bizarre workarounds, waiting to fall apart any day.


If you really want to start worrying, consider how the respective scsi.device RDB handling might slip up. In the small set of tests I made so far I found that disk drivers as a rule do not handle storage devices well which use sector sizes larger than 512 bytes.

The RDB specs call for sector sizes larger than 512 bytes to be supported, but the code I have seen may fail to find the RDB header block because it may not check for larger sector sizes.
 

Offline olsen

Re: Hyperion announces OS 3.1 update
« Reply #25 on: December 30, 2017, 01:43:27 PM »
Quote from: Gulliver;834484
Validation: The mother of all evils
Only for file systems designed in around 1978, upgraded in 1987, which suffer from "scalability" limitations ;)

Quote
One of the major setbacks AmigaOS 3.1.x users will face, is that along with the benefits of large drives and newer FFS support, big volumes will definately become very common. This means that if you had to wait for quite some time before to get your volumes validated, on lets say a 500MB partition, just imagine how long will it take with a 20GB partition (and I am being really modest here).
The FFS is not well equipped to handle volumes that large.

For example, the file system startup itself will take a very long time, because once a volume is mounted, the OFS/FFS/DCFS mode requires that all the currently allocated blocks on the disk are counted. The file system literally keeps counting bits.

File sizes beyond 2,147,483,647 bytes are not entirely safe to use although in theory they might just work if you only read such files sequentially. More trouble begins if you write more than 4,294,967,295 bytes to a file. This might actually work, but the byte counters in the file header block will roll over which breaks the relationship between the reported file size, the number of blocks used as reported, the number of blocks actually used by the file. Disk repair utilities might just truncate such files during the repair operations.

Finally, the file system cannot manage volumes larger than 4,294,967,295 blocks, and it might not even know that it can't handle more than that.

The validation process itself is "just" one problem here, and by comparison, it only gives you trouble rarely, whereas the limitations of the daily file system operations are giving you trouble on a much more permanent basis :(

Quote
I know, clever users will still partition drives in a more reasonable manner, so that volumes dont get that large, but then is that a real solution?
Given the constraints (creaky design which did not evolve and improve over time, practically impossible to fix), this is what users tend to do. They cope by limiting what they can do with the file system and the media the file system uses. This started way back when Amiga hard disk drives became much larger than the comparatively "safe" 4 Gigabytes. If memory servers, that was in 1996.

Quote
And please dont tell me this is no problem, as this has been happening for decades, despite all AmigaOS developers not visualizing it as an issue.

Of course it's a problem, it's just quite difficult to fix. Actually, I have a cunning plan how to add journaling with write-ahead logging to the FFS in a backwards-compatible manner which will resolve the validation issue (well, up to a point). This is a long-term project, though. At the moment I'm working on something else that deals with file system defect repairs and data recovery.

Quote
My proposed solution is a cli command to tweak FFS behaviour:

C:Validate

Changes how the filesystem manages write errors.

OPTIONS

oneatatime -Validates/fixes volumes one at a time.
novalidate -Ignores any FS errors and doesnt validate/fix them.
prevent    -Attempts to trap the reset signal so that FS write corruption may be prevented.
warn       -Detects FS errors and gives a warning but doesnt perform any repair/validation action at all.
fix        -This is the default FS behaviour: If FS error occurs, all volumes are mounted read only and validate/repair starts on all of them.
waitfix    -If FS error occurs all volume activity is halted until validate/repair is finished. Then volume activity resumes as usual.
ask        -If error in the FS occurs, the user is asked to either validate/repair, ignore or mount readonly.

some options can be combined:

the "prevent" option can be combined with any other one.
"oneatatime" can be combined with "waitfix", "ask" and obviously as mentioned before, with "prevent".

I'd say that this would be doable from a technical point of view. Not so much from the point of view of available manpower to make it happen right now, as far as I can tell :(
 

Offline olsen

Re: Hyperion announces OS 3.1 update
« Reply #26 on: December 31, 2017, 09:02:54 AM »
Quote from: LoadWB;834511
Does your cunning plan involve radishes?


Definitely not ;)

The difficult part is making the journal operations work within the framework of the Amiga file system and stay backwards compatible.

There is a precedent for this kind of thing in NetBSD, for example, in the form of the WAPBL feature (WAPBL stands for 'write-ahead physical block logging'). WAPBL can keep a log for a file system separate from the file system's blocks or it can keep it on the file system itself.

The type of journaling I have in mind would be limited to metadata blocks, and it would log complete blocks rather than "compress" the individual metadata changes (e.g. change a file name, change the file size, etc.) and store them. I read a paper by the principal developer of the ext3 file system (Theodore Ts'o) in which made the convincing point that logging complete blocks can be safer than logging "compressed" metadata changes. After all, if the block which individual metadata changes are applied to already has defects, the changes will probably make things worse. Logging complete metadata blocks has been shown to do better in this respect.

Anyway, this kind of journaling could be added to my 'C' language FFS reimplementation with very little changes required. However, making the journal operations fast is tricky, and I also still have to figure out how large the journal should be. This could be an interesting project for 2018 :)

(Incidentally, did anybody actually like series 1 of Blackadder, or is it just that it pales so much in comparison to the series which followed it?)
 

Offline olsen

Re: Hyperion announces OS 3.1 update
« Reply #27 on: December 31, 2017, 11:21:33 PM »
Quote from: kolla;834539
Considering the large number of Amiga FFS implementations, one would think it's a well understood filesystem.


How the file system data structures look like and work together is reasonably well-understood. One might still wish for some of the unexplainable mysteries to be resolved (e.g. why are the data block references in a file header processed in reverse order, why does the root block indicate the size of the hash table when this property follows directly from the size of the block), but you generally get a pretty good idea of how the parts hang together.

Still, that isn't the whole story. The implementation language and its operating system set a few rules which you are supposed to know, except nobody really wanted to explain them to you in the early days of the Amiga. Things didn't get much better, documentation-wise, in the decade that followed, save for Ralph Babel's "Amiga-Guru book" which succeeded in explaining everything that could be explained. The file system design assumes that the basic building block for all data structures, the 32 bit word, is a signed quantity, but both design and implementation tend to ignore this. Put another way: the design is generally unaware of its limitations, and so is the implementation. This is why you should be worried if you are about to deal with files larger than 2,147,483,647 bytes. You cannot necessarily predict what's going to happen. Same thing with the volume size, for which strange things might just happen if more than 2,147,483,647 blocks are used.

It does not at all look sunny if this file system has to run on the Amiga operating system, because it has to interface with the dos.library and application software which uses the dos.library API and/or the packet interface. What is often overlooked is just how complex the file system implementation has to be, because it has schedule the packet I/O and treat each client fairly. Since dos.library doesn't do file systems any favours, what you end up with in the FFS, or rather the default ROM file system, is almost a self-contained multitasking operating system of its own.

The original BCPL version did lean on the BCPL runtime library so as to provide for coroutine threading (think Python generators: this is the closest thing I can think of in terms of how they work). The assembly language versions which followed had to implement their own version thereof, and as the file system functionality became more complex, it became harder and harder to guarantee that certain operations actually worked out correctly.

For example, writing to a file can trigger a series of operations which all have to complete in a specific order and need to be reversible in case of trouble: the file may have to be extended, for which new storage space will have to be allocated, the storage space needs to be linked up with the bookkeeping data structures which also may have to be reallocated, and then everything has to be recorded in the file header, and eventually the client will need to be notified that the write operation succeeded. Unless something goes wrong, of course, in which case the whole cascade will have to be rolled back. And don't forget that during the rollback errors might occur.

All of this uses strange data structures which don't always suffice for carrying around all the information that is needed. Weird workarounds exist, such as for stuffing everything that doesn't fit into a file handle or the "file info block" into the humble file lock, which then has interesting repercussions all over the place. It's not just that file locks become "relay batons" for multi-stage operations, the file system has to watch out for race conditions (oh, the irony), too.

This threading model is one reason why the file system is so brittle, and desaster might follow if the "relay" is aborted (crash, disk ejected, etc.). The file system design generally does not care in which order the operations that modify its data structure are completed, but the implementation should. The FFS version 40 tries its best to modify the disk data structures in an order which minimizes the irreversible damage should one modification go awry. But this only goes so far: the Amiga file system has a write cache which tends to aggravate any problem that may occur in this domain.

I've seen the belly of the beast from the inside when I reimplemented the FFS in 'C'. This was, in retrospect, the toughest software project I have ever undertaken. It certainly is the most complex piece of software I have written so far.

That's what you get from seemingly simple designs...
 

Offline olsen

Re: Hyperion announces OS 3.1 update
« Reply #28 on: December 31, 2017, 11:31:58 PM »
Quote from: LoadWB;834548
Sun implemented journaling in UFS in Solaris 7.  I am curious how it did so but I have never spent the time to dig up the information.

If I remember correctly, the NetBSD WAPBL feature did not require the file system to cooperate much. You could implement it as an abstraction layer on top of the low level block access. It probably did call for a more optimized, higher performance journaling implementation.

Quote
I liked it, but I think I enjoyed the first season more for Brian Blessed.

Now that you mention it... It's hard to think of Brian Blessed not making the kind of contribution which elevates the end result far beyond what it might otherwise have deserved.
 

Offline olsen

Re: Hyperion announces OS 3.1 update
« Reply #29 from previous page: January 03, 2018, 03:39:58 PM »
Quote from: Gulliver;834620
@Thomas Richter
@olsen

A journaling filesystem woulld certainly improve reliability.

But why not better kill more birds with one stone?

A versioning filesystem would be a much better overall solution, because not only it would allow the user to roll back to previous versions of files in case of system corruption, but it also means that if the filesystem does the work twice (comitting to disk), you at least do the job completely and not halfway like in a journaled filesystem, and also keep the benefit of having an older backup automatically.

The drawback, is of course, increased disk space usage, but is that really a problem in an Amiga system where files are really small in size and when storage media keeps getting bigger each day? And of course, you can also provide some algorithm for garbage collection when the amount of backup exceeds certain amount to free some space.


I don't quite see this yet. There is not a lot of wiggling room within the constraints the FFS on-disk data structures impose upon you. With a log of changes growing as write operations take place, you will have to deal more and more with managing disk space. This is where this file system design is really weak. There are no data structures to assist making optimal storage allocations under certain constraints (e.g. allocate closest to the tail end of the file, allocate closest to the file header block, pick the largest consecutive chunk of free space, etc.). As far as the FFS is concerned, the storage space is broken down into two big sets of almost identical size with no block preferable to the other.

If I remember correctly how log-structured file systems work, I can see how they are more attractive for use with SSDs. You rarely rewrite anything, if at all. The downside is that you have to deal with accessing the versions (through the existing file system API? we can't conveniently upgrade dos.library), representing the available storage space (minus the versioned data which could be reclaimed) and with scrubbing the log. You have to have all three, and each of these is quite challenging to implement.

By comparison, journaling with write-ahead logging would be much simpler to implement. It could be integrated into the metadata block access operations and the remainder of the file system, its APIs and data structures would remain unchanged.

I'd rather prefer to tinker with something that I understand up to a point, and which is extensible, than having to invent a big bag of new stuff, all of which would need a lot more testing just to make sure that it works as intended, including how it deals with defects.