Welcome, Guest. Please login or register.

Author Topic: The Os 3.1.4 Thread  (Read 247831 times)

Description:

0 Members and 10 Guests are viewing this topic.

Offline olsen

Re: The Os 3.1.4 Thread
« on: September 27, 2018, 05:03:15 PM »
Did you or Olaf speak to Detlef Würkner (tetisoft)?

Not to my knowledge. Porting code from OS4 is a thorny issue indeed.

Some of the earlier work on OS4 (2003-2007) was still built and tested on 68k machines before it was ported and developed further on PPC hardware. This is code which is both useful or interesting and likely to port easiest, but it is also code which has not been touched in more than 10 years. I still believe it would be a good idea to look at what we have in the archives, but code review and rework are guaranteed to follow. As small as the team was, this was not always an option.

Another constraint results from the code ownership. We can only work with material for which the author and owner consents that it may be adapted. So far we have limited ourselves to material for which the author's consent had already been granted or was easily obtained. Please do not assume that the "easy way" was at all easy in the end. It was "just" a trade-off in terms of time and manpower. Even something seemingly simple such as the whole set of tools and drivers for mass storage devices (e.g. scsi.device, HDToolbox, ProdPrep) quickly ballooned and required that we seek advice before finding an acceptable solution to apply fixes and enhancements. This kind of thing happened again and again over time. And the HDToolbox saga, for example, still is not over -- some of the work we began has in fact turned into research projects.

If anything, the Amiga operating system and its components are, or have become over the course of a decade, more demanding of finding clever ways to resolve their shortcomings, or to find robust workarounds.
« Last Edit: September 27, 2018, 05:06:11 PM by olsen »
 

Offline olsen

Re: The Os 3.1.4 Thread
« Reply #1 on: September 28, 2018, 07:17:32 AM »
Right. I'm aware of that and obviously am aware of the code ownership restraints. You are to be commended for finding workarounds.

I only mentioned/linked to detlef because at that time we seem to have encountered similar issues to what you are discussing...lack of sources, working with a binary only, issues with prefs, etc. I just happen to see a lot of parallels in the struggle.
I thought perhaps if all of detlef's postings from that era were examined, you might find something helpful, or perhaps an issue not considered yet.

#6

Personally, I feel uncomfortable to "mine" the work of other developers for clues instead of asking them directly, if they are still around and are inclined to go over their past work, offering insights. Ten, twenty or more years down the line it does become difficult to remember the rationale for design and implementation decisions made, though.

Old code tends to make you make the worst assumptions about the motivations and especially the ability of the original developer and designer (exceptions exist, though: I still find Exec to be exceptionally slick code, for example, and just to mention it, the Amiga platform-specific code of the Amiga Unix kernel). You're almost always biased, and it takes an effort to dial back how harshly you are bound to judge what you will see. Well, you are looking for bugs after all, and compatibility/portability issues, too. But you cannot conveniently infer the reason behind the design and implementation. What works for me is to remind myself constantly that the guys who wrote this weren't "stupid", it's just that I don't know enough about the code yet: I'm the one who's stupid right now.

The code does not always provide answers, and given its age, the only way forward often is to go back to the specifications for which it was built, e.g. the ROM kernel reference manuals, schematics, etc., and do your homework ;) If you are lucky, you can talk to other developers who have been around a long time and who can provide context. We have been very lucky in this respect (and here's hoping dearly that our luck will not run out any time soon).

That said, nothing beats reading the code and testing how it works and interacts. This is what takes so much time, especially in such a small team. The printer.device is one such case. We lacked the source code to the device as used in the OS 3.5/3.9 updates, so for the OS4 work I "ported" the V39 code to build cleanly and portably under SAS/C, then moved it to the PPC platform. Detlef Würkner's work would subsequently build upon it.

The cleanest way to start over for the 3.1.4 update seemed to be to take the SAS/C port I had prepared and take it from there. Sure enough, this was 10-15 year old code which had not been reviewed since. There was trouble in every corner. Thomas reviewed the printer.device, tested, updated and rewrote it, filling in the gaps in the implementation to allow for OS 3.5/3.9 driver and prefs compatibility. Then it turned out that the HP printer drivers which were added in the OS 3.5/3.9 update were just as lacking as the printer.device and had to be reviewed, tested and updated. This was pretty scary :(

I'm still floored by what Thomas accomplished. At the back of my mind I still do wonder if printer.device deserved that much love, though. It's an interface for transforming bitmap imagery and for producing text output on mechanical devices which, unless you exclude the generic PostScript and HP-PCL printer drivers, are now so old that they are no longer being manufactured. Still, this is probably part of the package if you are going to update an operating system and its components which was last updated at this scale some 24 years ago. Funny thing: the reconstruction of Babbage's Analytical Engine includes a print output device (as part of the original design), so who am I to argue ;)
« Last Edit: September 28, 2018, 08:23:09 AM by olsen »
 

Offline olsen

Re: The Os 3.1.4 Thread
« Reply #2 on: September 28, 2018, 04:36:42 PM »
@olsen

Quote
The cleanest way to start over for the 3.1.4 update seemed to be to take the SAS/C port I had prepared and take it from there. Sure enough, this was 10-15 year old code which had not been reviewed since. There was trouble in every corner. Thomas reviewed the printer.device, tested, updated and rewrote it, filling in the gaps in the implementation to allow for OS 3.5/3.9 driver and prefs compatibility. Then it turned out that the HP printer drivers which were added in the OS 3.5/3.9 update were just as lacking as the printer.device and had to be reviewed, tested and updated. This was pretty scary :(

This really is baffling to me and it seems to me a mission anal retentiveness rather than to be useful in the real world!!! The Printer Drivers / printer.device were ALWAYS rubbish and are not worth updating!

The code never significantly evolved out of the state which the Workbench 1.3 update dropped it into. As the printer market evolved during the six years which followed Workbench 1.3 Commodore never caught up. This was likely just the kind of expensive development work which Commodore was "famous" for not undertaking.

I don't think that the printer.device or the drivers were always rubbish. For a brief time (say 1986-1988) they were adequate, but both the driver design and how the text and graphics output shaped the design of the printer.device stopped it from ever being able to become relevant again. Apple, for example, early on settled on a graphics device abstraction layer which both worked for the display and the printed output (the original Quickdraw even tried to encompass component colour printing, which was eventually replaced when proper RGB colour support came to the Macintosh II). For the Amiga it was bitmap colour graphics all the way, within the constraints of the original custom chipset, slightly updated in Workbench 3.0.

In any case, custom printing software which clung to the printer.device API was just as constrained by the limitations of the architecture as the original printer.device. The big problem lies with how "printing" was designed for the Amiga first, and with the limitations of the implementation second.

Quote
Everyone who wanted to use a printer on their classic who didn't want to be limited to a centronix era dot-matrix monstrousity bough TurboPrint period. There is absolutely NO point reinventing the wheel just because TurboPrint is no longer developed just as in the same way OS3.9 is no longer developed. These products were good and can still be bought / downloaded the last time I checked!

Maybe people just love putting right what once went wrong in a weird geeky 'Quantum Leap' kinda way but this project really doesn't seem to provide ANY real world benfits to the few Classic users remaining over and above what has gone before.

That certainly depends upon your expectations, and what we could deliver. For example, with all the bug fixes and enhancements, there's a new file system recovery tool included which to the best of my knowledge has never existed before. It may yet be useful...

Quote
OS3.9 IS STILL FOR SALE AT VESALIA'S WEB STORE!!!

Is it? How come that new copies are still available after some 20 years? I do wonder how this is possible.

Quote
THOMAS ET AL. ARE COMPETING AGAINST A 16 YEAR OLD PRODUCT

Well... not really. The scope of the OS 3.9 update is different from what we came up with. Fixing the printer.device and drivers was an unexpected detour, but it is not representative of the OS 3.1.4 update as a whole. There are bug fixes in it which which were not even in reach for the OS 3.5/3.9 update, and for that matter, for Kickstart/Workbench 1.3.

The major reason why the work was done is in that it makes more sense to adapt the operating system than to require that the designers and developers of the new Amiga hardware which has become available as of late to adapt their hardware instead. There's a towering stack of unfixed bugs and increasingly rickety workarounds in place in the Amiga operating system and, for example, Picasso96. It's about time that this is addressed.

Quote
AND DON'T SEEM TO CARE ABOUT IMPROVING ON IT (FEATURES, COMPATIBILITY AND SPEED IS ALL MOST OF US REALLY CARE ABOUT). IT SEEMS THOMAS MIGHT BE TEMPTED TO COMPETE WITH OS3.9 IF WE ALL INVEST IN THIS SPELLING AND GRAMMAR CHECKING EXCERCISE FIRST!! IS THAT ABOUT THE LONG AND THE SHORT OF IT? AM I MISSING ANYTHING?

The satisfaction of setting something right that was broken and could be repaired :) Yes, really: leave things better than you found them.
« Last Edit: September 28, 2018, 04:58:43 PM by olsen »
 

Offline olsen

Re: The Os 3.1.4 Thread
« Reply #3 on: October 04, 2018, 11:28:21 AM »
Very nice! I'm curious if you could tell us more about testing: do you use test frameworks? What test cases exist? Did you inherit test cases or build your own test suites?

Cheers!

The Amiga operating system and all its components matche one of the textbook definitions of legacy code: code for which no automated tests exist. It is not designed to be testable, which makes it very hard to both design effective tests and to deploy a testable version in the same context as it would be in if it were production code and not test code (ROM space is the major limitation here). The operating system source code is a mix of 'C' and assembly code, both of which provide very little leverage to achieve abstraction layers which are much more easily available in object-oriented languages.

If you are used to testing frameworks to make your life easier, you know what you are missing. I did investigate how testing of complex 'C' code is being done, and the sad result seems to be that is an unsolved problem. The testing frameworks I found which apparently catered to the need of automated 'C' code testing seemed to be not much more useful and powerful than the equivalent to planting assert() calls and the occasional printf() in the code (debugging like it's 1983), so I was none the wiser.

As far as I can tell there is no worthwhile test framework for 'C' legacy code, but I'd very much like to be proven wrong on that... So we did not use one on AmigaOS, which is why there are no use cases and no test suite to speak of.

Compatibility testing still worked very much like it did when Commodore was still in business: testing by exercising the features, discovering compatibility issues more or less by chance as well as by methodical exploration.

One exception is the new "Disk Doctor" which I wrote from scratch for Workbench 3.1.4. The code was designed to be testable, and it even includes a damage simulation feature which makes the disk access layer pretend that certain forms of data corruption or read errors exist on the medium. Automated testing involved having Disk Doctor examine and recover data from hundreds of Amiga disk images.
« Last Edit: October 04, 2018, 11:47:13 AM by olsen »
 

Offline olsen

Re: The Os 3.1.4 Thread
« Reply #4 on: October 04, 2018, 11:43:04 AM »
But I still think this is mainly about stopping Vampire users pirating core OS elements for VampireOS and instead encourage them to pay Hyperion for the pleasure.

Do you really think we wasted thousands of hours of spare time to achieve just that?

They built the Eiffel tower to investigate how well the corrosion-proof coating of the cast-iron tower endures the harsh conditions at 320m elevation, near its tip, didn't they? ;)
 

Offline olsen

Re: The Os 3.1.4 Thread
« Reply #5 on: October 04, 2018, 04:26:06 PM »
Anyone know why the Hyperion page still charges VAT for those that live outside the EU?

Source

Only Hyperion's "fullfillment partner" will respond to that apparently.

#6

Not exactly an explanation, but I recently learned how complicated the VAT issue has become for services rendered and goods delivered both within the European Union and from the European Union to non-member states. In my opinion it is quite likely for a payment processor such Avangate to slip up here.

Within the European Union the VAT added on top of the net price used to be the rate current in the country from which it was sold. For example, with the seller in Germany the standard rate of 19% VAT rate would be applied to a customer in Denmark. This was changed recently, and now the VAT rate of the customer has to be applied to the net price, e.g. for a seller in Germany and the customer in Denmark, the standard rate of 25% would have to be applied. Sounds sufficiently complicated, and it certainly is (and gets more so because you have to keep track of the specific payments made and the taxes you charged on top of the net price). It gets more complicated still because of exceptions the individual countries apply to taxation. Luckily, these tend not to affect services and goods rendered through payments made through online service because this has been standardized.

It gets really tricky when you sell from the European Union to a non-member state such as Switzerland or Norway. Services and goods may be taxed differently or not at all, for example. So the buyer may have to pay import taxes on floppy disks and EPROMs, but not on the digital download which contains the same information as the floppy disks and EPROMs (or it might be a reduced tax rate).

There's plenty of opportunity for things to go wrong.
« Last Edit: October 04, 2018, 04:54:37 PM by olsen »
 
The following users thanked this post: nicholas

Offline olsen

Re: The Os 3.1.4 Thread
« Reply #6 on: October 05, 2018, 03:56:31 PM »
Is it possible to make a Super Kickstart Disk with the A3000 ROM images using your tool Olsen?


http://aminet.net/package/util/misc/MakeSuperDisk

Yes, but some assembly is required.

Both the 1.3 and 2.x (and by extension the 3.x) ROMs on a SuperKickstart disk consist of both the respective ROM image (256 KB for Kickstart 1.3, 512 KB for everything else) and what's called "bonus" code which is appended to the ROM. The Kickstart selection menu of the A3000 boot roms checks if the bonus code is present, and if it's not it will refuse to load the respective Kickstart ROM image.

The purpose of this specific "bonus" code (there's a bonus module in the A3000 ROM which has a different purpose) is to set up the MMU and exception handlers which allow for the respective Kickstart loaded to remain in memory until you switch it off. It's a software solution for what the Amiga 1000 did in hardware back in the day ;)

You can make the AmigaOS 3.1.4 A3000 Kickstart ROM suitable for use on a SuperKickstart disk by copying the respective bonus code from what's in your A3000 "WB_2.x:devs/kickstart" file and appending it to the Kickstart ROM image. If the "WB_2.x:devs/kickstart" file is larger than 524288 bytes, then copy all the data which begins at the 524289th byte until the end of the kickstart file. That data contains the bonus code which the Kickstart selection menu expects. Append this bonus code to the AmigaOS 3.1.4 A3000 ROM file and the MakeSuperDisk program should be able to use it.

Come to think of it, we could add the A3000-specific bonus code to the ROM image and include it in the A3000 download package as a "SuperKickstart ROM image" file.
« Last Edit: October 05, 2018, 04:00:15 PM by olsen »
 
The following users thanked this post: nicholas

Offline olsen

Re: The Os 3.1.4 Thread
« Reply #7 on: October 15, 2018, 05:12:18 PM »
Hi Thomas and Olsen!

Thank you for your answers, very interesting!

...

I don't know of a good framework to test C code either, even less so for OS and OS ROM and libraries... Could you explain in more details what you mean about "It is not designed to be testable"? Naively, I would have think that, being a set of libraries (in ROM or software), there could be tests for each and every available data structures and functions provided by these libraries, but I'm certainly missing something here... in particular, when it comes to dynamic stuff, like tasks and processes.

What is missing in the operating system in general is even the most humble unit test which more complex tests could build upon.

From how I understand the rationale for implementing tests (as described in great detail in Michael Feathers' book "Working effectively with legacy code"), the tests are to assist in making code changes more robust, if not allowing them to be made in the first place. Code which is more easily changed is also likely to be more easily changed for the better: reduced complexity, fewer side-effects, better understanding of its design and relationships within the context it is a part of.

There is nothing there which one could build upon. You would have to start from scratch and construct the unit tests as well as build test harnesses. The Amiga operating system is comparatively small but it is by no means simple. Adding the scaffolding for building a testable operating system out of what we have is a major challenge.

The "modern" testing approach which grew out of the use of object oriented implementation languages was not available to the designers of the Amiga operating system. This is why testing began from the outside in, by testing how software of particular interest/import interacted with the operating system, and by analyzing the behaviour triggered by the changes which were introduced. It might have been the industry standard in the 1980'ies and the 1990'ies.

What methods I applied under such circumstances (only 'C' and assembly language used as implementation languages) came from the books I read on this matter: "Writing solid code" (Steve Maguire), "Code complete" (Steve McConnell) and "Clean code" (Bob Martin). Unit testing is part of this toolbox, but it's much easier to write new code to follow the principles described in the books than to make legacy code conform to them after the fact. The new Disk Doctor was written from the ground up and this is why I could design it to be testable, and to apply the best practices I learned from these three books (as well as I was able).

Quote
Could you tell us more about the features that you exercised? Do you have a set of programs that you know are "tough" on the OS and "play" with them to test the OS?

As far as I know "empirical testing" was used to as large a degree as was possible. You install the new software then try to use the programs you are familiar with. If necessary, unwelcome changes in behaviour are then documented and this goes into the bug tracker. Then it's the old reproduce, analyze, fix, retest loop. There was no shortage of known operating system bugs which went through the same loop without having to start out as a bug tracker ticket. In short, the whole process used the methods available at the time when the Amiga operating system was designed and still being maintained in the years 1985-1994.

With the new Disk Doctor things were a bit different, and only because I had the luxury to start from scratch. The central data structure which the new Disk Doctor uses to keep track of what type of information it finds in a block on the medium had to be both very memory-efficient and still fast enough to work on a plain 68000 machine. To this end I designed and implemented a sparse bit array, with unit tests and a test harness. This was tested separately before I integrated it into the Disk Doctor code. Testing the new Disk Doctor both involved old school "empirical testing" and building a test set of some 450 ADF images which were then fed into the new Disk Doctor through a script file. This test exercised the entire program, including its in-memory database, the memory management, and the diagnostic as well as the data recovery functions.
 
The following users thanked this post: Tygre

Offline olsen

Re: The Os 3.1.4 Thread
« Reply #8 on: December 26, 2018, 02:06:36 PM »
Note that "HDToolBox" keeps a database of disks. You can always "add new drive type" and there trigger a re-scan of the harddisk, then use this as basis for new partitions. Otherwise, HDToolBox will use whatever it saved the last time it recognized the drive.

Huh, I never knew that. Is that something new?

No, this is a very old feature which goes back to the days of the A590 which shipped with 3.5" XT drives (ST 506 interface, ~20 MB per drive), but would also accept a SCSI drive.

The problem which the database solved was that the XT drives did not necessarily report their respective model and properties correctly when prodded. Commodore shipped a database of known-working drive types and their properties with the HDToolBox program, with the option for the user to add more drive types by entering the respective configuration data manually. For example, if you flipped over the A590 case, you would see a label describing the type, manufacturer and model, along with the number of cylinders/sectors/tracks/heads. If you ever lost the database, or corrupted it by accident, you could at least set up your drive correctly by following the label data. The usual approach would be to pick a database entry which matched the model and manufacturer information.

The problem with hard disk drives not producing reliable information about their respective configuration persisted for a very long time indeed, right until into the mid-1990'ies before the great mergers/acquisitions of hard drive manufacturers left us with only a handful of companies (Seagate, Western Digital, Fujitsu, Toshiba).

There is some extraordinarily complex code in HDToolBox which tries almost every trick that might work for XT and SCSI drives of that age to figure out the configuration parameters. To those unfamiliar with its purpose the code is perplexing and outright impenetrable. When we updated HDToolBox for the 3.1.4 update, we asked around for help and received kind advice from Ralph Babel and Joanne Dow. When Joanne kindly showed us how the RDPrepX tool approached the same problem as the mystifying HDToolBox configuration code, the penny finally dropped and the pennies just kept dropping. This kind of code is supposed to be as complicated/peculiar as it appeared :o

Long story short: the database in HDToolBox and the arcane configuration parameter heuristics have no place in a (somewhat) modern hard disk setup program. But for the time being we will be stuck with it :(
« Last Edit: December 26, 2018, 02:34:11 PM by olsen »
 
The following users thanked this post: Tygre