Welcome, Guest. Please login or register.

Author Topic: Hyperion announces OS 3.1 update  (Read 90860 times)

Description:

0 Members and 3 Guests are viewing this topic.

Offline olsen

Re: Hyperion announces OS 3.1 update
« Reply #194 from previous page: December 31, 2017, 09:02:54 AM »
Quote from: LoadWB;834511
Does your cunning plan involve radishes?


Definitely not ;)

The difficult part is making the journal operations work within the framework of the Amiga file system and stay backwards compatible.

There is a precedent for this kind of thing in NetBSD, for example, in the form of the WAPBL feature (WAPBL stands for 'write-ahead physical block logging'). WAPBL can keep a log for a file system separate from the file system's blocks or it can keep it on the file system itself.

The type of journaling I have in mind would be limited to metadata blocks, and it would log complete blocks rather than "compress" the individual metadata changes (e.g. change a file name, change the file size, etc.) and store them. I read a paper by the principal developer of the ext3 file system (Theodore Ts'o) in which made the convincing point that logging complete blocks can be safer than logging "compressed" metadata changes. After all, if the block which individual metadata changes are applied to already has defects, the changes will probably make things worse. Logging complete metadata blocks has been shown to do better in this respect.

Anyway, this kind of journaling could be added to my 'C' language FFS reimplementation with very little changes required. However, making the journal operations fast is tricky, and I also still have to figure out how large the journal should be. This could be an interesting project for 2018 :)

(Incidentally, did anybody actually like series 1 of Blackadder, or is it just that it pales so much in comparison to the series which followed it?)
 

Offline Kronos

  • Resident blue troll
  • Hero Member
  • *****
  • Join Date: Feb 2002
  • Posts: 4017
    • Show only replies by Kronos
    • http://www.SteamDraw.de
Re: Hyperion announces OS 3.1 update
« Reply #195 on: December 31, 2017, 10:54:31 AM »
Quote from: olsen;834531

(Incidentally, did anybody actually like series 1 of Blackadder, or is it just that it pales so much in comparison to the series which followed it?)


The later series were just repetitive nonsense, sure very good repetitive nonsense but still just repetitive nonsense ;)

1st series had some sort of "sensible" storyline.
1. Make an announcment.
2. Wait a while.
3. Check if it can actually be done.
4. Wait for someone else to do it.
5. Start working on it while giving out hillarious progress-reports.
6. Deny that you have ever announced it
7. Blame someone else
 

Offline kolla

Re: Hyperion announces OS 3.1 update
« Reply #196 on: December 31, 2017, 05:04:53 PM »
Considering the large number of Amiga FFS implementations, one would think it's a well understood filesystem.
B5D6A1D019D5D45BCC56F4782AC220D8B3E2A6CC
---
A3000/060CSPPC+CVPPC/128MB + 256MB BigRAM/Deneb USB
A4000/CS060/Mediator4000Di/Voodoo5/128MB
A1200/Blz1260/IndyAGA/192MB
A1200/Blz1260/64MB
A1200/Blz1230III/32MB
A1200/ACA1221
A600/V600v2/Subway USB
A600/Apollo630/32MB
A600/A6095
CD32/SX32/32MB/Plipbox
CD32/TF328
A500/V500v2
A500/MTec520
CDTV
MiSTer, MiST, FleaFPGAs and original Minimig
Peg1, SAM440 and Mac minis with MorphOS
 

Offline Oldsmobile_Mike

Re: Hyperion announces OS 3.1 update
« Reply #197 on: December 31, 2017, 05:15:05 PM »
Quote from: kolla;834539
Considering the large number of Amiga FFS implementations, one would think it's a well understood filesystem.


We have an awful lot of file systems for such a small market. ;)
Amiga 500: 2MB Chip|16MB Fast|30MHz 68030+68882|3.9|Indivision ECS|GVP A500HD+|Mechware card reader + 8GB CF|Cocolino|SCSI DVD-RAM
Amiga 2000: 2MB Chip|136MB Fast|50MHz 68060|3.9|Indivision ECS + GVP Spectrum|Mechware card reader + 8GB CF|AD516|X-Surf 100|RapidRoad|Cocolino|SCSI CD-RW
 Amiga videos and other misc. stuff at https://www.youtube.com/CompTechMike/videos
 

Offline kolla

Re: Hyperion announces OS 3.1 update
« Reply #198 on: December 31, 2017, 07:01:28 PM »
Quote from: Oldsmobile_Mike;834540
We have an awful lot of file systems for such a small market. ;)


Yeah, it's ridiculous.
B5D6A1D019D5D45BCC56F4782AC220D8B3E2A6CC
---
A3000/060CSPPC+CVPPC/128MB + 256MB BigRAM/Deneb USB
A4000/CS060/Mediator4000Di/Voodoo5/128MB
A1200/Blz1260/IndyAGA/192MB
A1200/Blz1260/64MB
A1200/Blz1230III/32MB
A1200/ACA1221
A600/V600v2/Subway USB
A600/Apollo630/32MB
A600/A6095
CD32/SX32/32MB/Plipbox
CD32/TF328
A500/V500v2
A500/MTec520
CDTV
MiSTer, MiST, FleaFPGAs and original Minimig
Peg1, SAM440 and Mac minis with MorphOS
 

Offline LoadWB

  • Hero Member
  • *****
  • Join Date: Jul 2006
  • Posts: 2901
  • Country: 00
    • Show only replies by LoadWB
Re: Hyperion announces OS 3.1 update
« Reply #199 on: December 31, 2017, 07:54:46 PM »
Quote from: olsen;834531
Anyway, this kind of journaling could be added to my 'C' language FFS reimplementation with very little changes required. However, making the journal operations fast is tricky, and I also still have to figure out how large the journal should be. This could be an interesting project for 2018 :)

(Incidentally, did anybody actually like series 1 of Blackadder, or is it just that it pales so much in comparison to the series which followed it?)


Sun implemented journaling in UFS in Solaris 7.  I am curious how it did so but I have never spent the time to dig up the information.

I liked it, but I think I enjoyed the first season more for Brian Blessed.
 

guest11527

  • Guest
Re: Hyperion announces OS 3.1 update
« Reply #200 on: December 31, 2017, 08:17:15 PM »
Quote from: kolla;834539
Considering the large number of Amiga FFS implementations, one would think it's a well understood filesystem.

Yes and no. The structure of the file system is well understood and documented. This does not mean that the implementation is well understood - or at least, the assembly implementation is not. It's 20 year old legacy assembly code.

There are also a couple of implications of the file system and the interface towards the dos.library that raises quite a couple of "architectural questions", to put it this way. ACTION_EXNEXT is probably one of the hardest to implement requests of the filing system, and it interacts deeply with the rest of the file system if done correctly.

In particular the assembly version is completely multi-threaded, which means that an ACTION_EXNEXT can overlap with file removal, creation and renaming. If journaling has to be itnerleaved in all this game again, the equation does not become simpler.

The C version is not as highly multithreaded as the assembly version, and for this reason easier to maintain. It might also imply - but I haven't tested - that it does not perform quite as well in situations where you have multiple processes accessing the disk simultaneously.
 

Offline olsen

Re: Hyperion announces OS 3.1 update
« Reply #201 on: December 31, 2017, 11:21:33 PM »
Quote from: kolla;834539
Considering the large number of Amiga FFS implementations, one would think it's a well understood filesystem.


How the file system data structures look like and work together is reasonably well-understood. One might still wish for some of the unexplainable mysteries to be resolved (e.g. why are the data block references in a file header processed in reverse order, why does the root block indicate the size of the hash table when this property follows directly from the size of the block), but you generally get a pretty good idea of how the parts hang together.

Still, that isn't the whole story. The implementation language and its operating system set a few rules which you are supposed to know, except nobody really wanted to explain them to you in the early days of the Amiga. Things didn't get much better, documentation-wise, in the decade that followed, save for Ralph Babel's "Amiga-Guru book" which succeeded in explaining everything that could be explained. The file system design assumes that the basic building block for all data structures, the 32 bit word, is a signed quantity, but both design and implementation tend to ignore this. Put another way: the design is generally unaware of its limitations, and so is the implementation. This is why you should be worried if you are about to deal with files larger than 2,147,483,647 bytes. You cannot necessarily predict what's going to happen. Same thing with the volume size, for which strange things might just happen if more than 2,147,483,647 blocks are used.

It does not at all look sunny if this file system has to run on the Amiga operating system, because it has to interface with the dos.library and application software which uses the dos.library API and/or the packet interface. What is often overlooked is just how complex the file system implementation has to be, because it has schedule the packet I/O and treat each client fairly. Since dos.library doesn't do file systems any favours, what you end up with in the FFS, or rather the default ROM file system, is almost a self-contained multitasking operating system of its own.

The original BCPL version did lean on the BCPL runtime library so as to provide for coroutine threading (think Python generators: this is the closest thing I can think of in terms of how they work). The assembly language versions which followed had to implement their own version thereof, and as the file system functionality became more complex, it became harder and harder to guarantee that certain operations actually worked out correctly.

For example, writing to a file can trigger a series of operations which all have to complete in a specific order and need to be reversible in case of trouble: the file may have to be extended, for which new storage space will have to be allocated, the storage space needs to be linked up with the bookkeeping data structures which also may have to be reallocated, and then everything has to be recorded in the file header, and eventually the client will need to be notified that the write operation succeeded. Unless something goes wrong, of course, in which case the whole cascade will have to be rolled back. And don't forget that during the rollback errors might occur.

All of this uses strange data structures which don't always suffice for carrying around all the information that is needed. Weird workarounds exist, such as for stuffing everything that doesn't fit into a file handle or the "file info block" into the humble file lock, which then has interesting repercussions all over the place. It's not just that file locks become "relay batons" for multi-stage operations, the file system has to watch out for race conditions (oh, the irony), too.

This threading model is one reason why the file system is so brittle, and desaster might follow if the "relay" is aborted (crash, disk ejected, etc.). The file system design generally does not care in which order the operations that modify its data structure are completed, but the implementation should. The FFS version 40 tries its best to modify the disk data structures in an order which minimizes the irreversible damage should one modification go awry. But this only goes so far: the Amiga file system has a write cache which tends to aggravate any problem that may occur in this domain.

I've seen the belly of the beast from the inside when I reimplemented the FFS in 'C'. This was, in retrospect, the toughest software project I have ever undertaken. It certainly is the most complex piece of software I have written so far.

That's what you get from seemingly simple designs...
 

Offline olsen

Re: Hyperion announces OS 3.1 update
« Reply #202 on: December 31, 2017, 11:31:58 PM »
Quote from: LoadWB;834548
Sun implemented journaling in UFS in Solaris 7.  I am curious how it did so but I have never spent the time to dig up the information.

If I remember correctly, the NetBSD WAPBL feature did not require the file system to cooperate much. You could implement it as an abstraction layer on top of the low level block access. It probably did call for a more optimized, higher performance journaling implementation.

Quote
I liked it, but I think I enjoyed the first season more for Brian Blessed.

Now that you mention it... It's hard to think of Brian Blessed not making the kind of contribution which elevates the end result far beyond what it might otherwise have deserved.
 

Offline kolla

Re: Hyperion announces OS 3.1 update
« Reply #203 on: January 01, 2018, 12:06:09 AM »
Right, and NetBSD also implements "ADOS" filesystem, Linux has affs, AROS and MorphOS also have their implementations. One would think some sort of cooperation would be beneficial.
B5D6A1D019D5D45BCC56F4782AC220D8B3E2A6CC
---
A3000/060CSPPC+CVPPC/128MB + 256MB BigRAM/Deneb USB
A4000/CS060/Mediator4000Di/Voodoo5/128MB
A1200/Blz1260/IndyAGA/192MB
A1200/Blz1260/64MB
A1200/Blz1230III/32MB
A1200/ACA1221
A600/V600v2/Subway USB
A600/Apollo630/32MB
A600/A6095
CD32/SX32/32MB/Plipbox
CD32/TF328
A500/V500v2
A500/MTec520
CDTV
MiSTer, MiST, FleaFPGAs and original Minimig
Peg1, SAM440 and Mac minis with MorphOS
 

Offline tonyvdb

Re: Hyperion announces OS 3.1 update
« Reply #204 on: January 01, 2018, 08:17:17 PM »
So, my concern with an update to OS3.1 is compatibility. Would software and hardware still function properly like the old video toaster?
Amiga 2000HD Indivision ECS
Amiga 4000D towerised OS 3.1 and 3.9 on CF cards
Indivision AGA, Mediator 4000
Video Toaster 4000 Flyer v4.3 Millenium.
202gig of video drive space & 5gig audio.
 

guest11527

  • Guest
Re: Hyperion announces OS 3.1 update
« Reply #205 on: January 02, 2018, 12:55:02 AM »
Quote from: tonyvdb;834585
So, my concern with an update to OS3.1 is compatibility. Would software and hardware still function properly like the old video toaster?

That is, of course, the plan. There is nothing like "new fancy toys" in the making that would potentially break compatibility.
 

Offline Gulliver

Re: Hyperion announces OS 3.1 update
« Reply #206 on: January 02, 2018, 06:28:46 PM »
@Thomas Richter
@olsen

A journaling filesystem woulld certainly improve reliability.

But why not better kill more birds with one stone?

A versioning filesystem would be a much better overall solution, because not only it would allow the user to roll back to previous versions of files in case of system corruption, but it also means that if the filesystem does the work twice (comitting to disk), you at least do the job completely and not halfway like in a journaled filesystem, and also keep the benefit of having an older backup automatically.

The drawback, is of course, increased disk space usage, but is that really a problem in an Amiga system where files are really small in size and when storage media keeps getting bigger each day? And of course, you can also provide some algorithm for garbage collection when the amount of backup exceeds certain amount to free some space.

As a huge side benefit, a versioning filesystem will also be an excellent filesystem candidate for flash/ssd solutions. We dont have any filesystem or tools to deal with wear levelling on Amigaland. A versioning filesystem does not need to overwrite the same blocks of data (unlike a journaled one). And you can make each block/cluster keep a write counter to estimate the media life and take action when reaching a critical life cycle limit.

Another benefit, is that due to the nature of the versioning filesystem, it may be used, with little customization as a developer aid.
« Last Edit: January 02, 2018, 06:32:39 PM by Gulliver »
 

Offline LoadWB

  • Hero Member
  • *****
  • Join Date: Jul 2006
  • Posts: 2901
  • Country: 00
    • Show only replies by LoadWB
Re: Hyperion announces OS 3.1 update
« Reply #207 on: January 02, 2018, 08:58:28 PM »
Now about adding snapshots?  Running ufsdump against fssnap is a dream :)
 

Offline Oldsmobile_Mike

Re: Hyperion announces OS 3.1 update
« Reply #208 on: January 02, 2018, 09:45:29 PM »
Quote from: Thomas Richter;834596
That is, of course, the plan. There is nothing like "new fancy toys" in the making that would potentially break compatibility.

Russian proverb, "New is first enemy of old".  :lol:
Amiga 500: 2MB Chip|16MB Fast|30MHz 68030+68882|3.9|Indivision ECS|GVP A500HD+|Mechware card reader + 8GB CF|Cocolino|SCSI DVD-RAM
Amiga 2000: 2MB Chip|136MB Fast|50MHz 68060|3.9|Indivision ECS + GVP Spectrum|Mechware card reader + 8GB CF|AD516|X-Surf 100|RapidRoad|Cocolino|SCSI CD-RW
 Amiga videos and other misc. stuff at https://www.youtube.com/CompTechMike/videos
 

Offline kreciu

Re: Hyperion announces OS 3.1 update
« Reply #209 on: January 02, 2018, 09:52:49 PM »
I would rather see work progress on AmigaOS4.2 and finished AmigaOS4.1 for looooooooooooooooooooooong  awaited A1222.

How long it takes to make a network driver? 3 years?
Re-A1200inE/BOX/3.2/AmigaOS3.2/TF1260@66Mhz/256Mb/MediatorTX/R9200SE/SpiderUSB/LAN/SB128/16Gb-CF/DVD-ROM/FDD-HD