Consequences of the AmigaOS 3.1 source code "leak", one year after?

[olsen]

You may remember that just about a year ago a file "amiga os source code 3.1.tar.bz2" popped up on a web site, was linked to, copied, and the contents even wound up on GitHub for a couple of days. This event was widely publicized, on Twitter, on personal blogs, and it even made the news.

That file would contain pretty much all the AmigaOS 3.1 source code, and plenty of other material which used to be available to Commodore developers back in 1994. It's safe to say that the contents of the archive are now very widely distributed, just not necessarily available to the general public.

Back then there was speculation as to who made the data available, where the data came from, and which consequences the availability would have.

It's been a year now, and I'm curious. What did the availability of the source code make possible?

(Careful: there could be legal strings attached to answering this question, so you might consider your options when posting answers here)

[olsen]

As a result we now have updated C:Sort and C:Wait programs in the "BB3+4", plus some other neatness (trackdisk.device 40.2?) right out of the "box". Especially the Wait update was useful, one can give a file as argument, and wait will sit there till the file shows up. Great! A lot of people have had fun looking at the sources and learning about the inner workings of whatever subsystem they are interested in - I could list names here, but I prefer they rather do it themselves (some have been quite open about on the Amiga Facebook groups), a lot of code has been improved because of this knowledge.

This sounds just like the positive outcome I thought might happen

Given how large that source code archive is, I expected that any change would happen at the edges: drivers, shell commands, etc.

[olsen]

Why? Copyright rests with the copyright holder until 70 years (probably longer) after expiration of the copyright holder.

It becomes Open Source 70 years after CBM filed for Chapter 11.

Hm... I think the expiration time for work created in this context is significantly longer.

As a single author (e.g. somebody such as H.P. Lovecraft, whose descendants and/or legal representatives did not renew the copyright for his works after his death), your work (currently) becomes part of the public domain some 70 years after your death.

For material created as work for hire, for a corporation (say, The Walt Disney Company), I recall that the expiration date was 112 years after it was created.

In any case, it is going to be a long time for this to happen. And even if it does, it may not have much of an impact any more. For example, the works of Johann Sebastian Bach and George Frideric Handel quickly lapsed into obscurity after they died, to be rediscovered and recognized some 100-150 years later.

[olsen]

So we are all hackers, users/developers... that all kind of blurs after a few decades.

It should.

As a hacker, the point of doing the work is often enough to "scratch an itch". The work starts small, with a specific focus. As time passes, the work grows and you can conveniently carry its design and application around in your head as it evolves.

More time passes, and looking back you may suddenly realize that you crossed a threshold at some point: instead of the "hack" you began with, you now have a system with numerous interdependencies.

That change implies a profound transformation of the work: you've left "hacker space" and entered "management space". One tends to be more fun than the other. Keeping a system in shape and operational means that you've joined the fire brigade whereas before you might have been a firebrand

[olsen]

LMAO we are sooo scared.

For some people it's a risk to be talking about the matter. I added the note so that you can make up your mind whether or not it's worth taking a risk in the first place.

Looks you already took that advice to heart - at least it got a laugh

[olsen]

If you have to ask that question, then there have been no consequences.

Either that, or they happened in places where you can't easily observe them. That's why I am curious: the current 68k Amiga operating system being what it is, resolving the small and big annoyances might have been enabled by the source code becoming available.

[olsen]

@Olsen
Is there any correlation between the leaked sources and the new 68k kickstart and wb updates?
Kamelito

No. The only relationship there is is in that both are based on the same operating system source code, last modified in May 1994.

The recent Kickstart/Workbench 3.1 changes use the unified native build which is the foundation for the AmigaOS 4 work. I dragged my 3.1 build out of storage in 2016 and converted it (again) from CVS format, to be usable with subversion instead.

[olsen]

Okay, no one has really come out and said it... but.. does it actually compile?

I seem to recall when it was leaked that people were having a hard time getting it to compile  anything, and that sources were missing.

Building the code is difficult, as most of it requires an Amiga-native compiler/assembler to create something that fits on the disk/into the ROM and does what it's supposed to. On top of that, hardly any two components of the operating system are built in the same manner, using the same set of makefiles and scripts.

Tinkering with the code is more like archaeology, when what you do involves sifting through layers of dust, ashes and earth before you can even begin to reconstruct the organization and systems as they were when they were still functional.

Can you compile it with gcc?

Not really.

[olsen]

@Olsen
Isn't it simpler in the long run to just backport AmigaOS 4.1 for Classic 68k Amiga without PPC?
It will surely be slower but as HW and Software simulation/emulation go faster in the long term it will be fast enough I guess and one code base is better than 2.

Kamelito

I don't believe that this would be the easier option, even in the long run.

The "threshold" when backporting code from AmigaOS4 to AmigaOS 3.x could still be accomplished with reasonable effort (time and manpower) was crossed in about 8-9 years ago. Ever since then new data structures and APIs have been added to AmigaOS4 which significantly increase the difficulties of backporting code.

This happened at Commodore, too, when Kickstart/Workbench 2.x was under development. The toolbox became larger (double the ROM space, double the leverage afforded to software developers), and it became more and more difficult for Commodore to provide the same tools to developers who wanted to use them both in 1.x and 2.x applications.

There weren't many such tools (I remember "amigaguide.library" and the Installer), and there were some 3rd party solutions such as a disk-loaded "gadtools.library". But inside the operating system, the new APIs and data structures created more tightly-coupled code, saving ROM space and allowing for more robust code to be written.

Backporting such code at some point means porting practically everything, because you cannot always conveniently resolve the new interdependencies. Even if you tried, you'd run into practical problems for a hypothetical AmigaOS4 for 68k: AmigaOS4 is designed for a platform with much more RAM. As these things go (Moore's law, etc.) it also requires a more powerful CPU to run smoothly than a 68k platform could deliver (unless you consider emulation a target platform that makes good business sense). Finally, that complex port would have to be tested as well, which is no small challenge to begin with.

[olsen]

in one sentence: os4 as a whole has become incompatible with amiga. which is actually the case for what i know. there is apparently less hassle to backport a good deal of aros software to genuine amiga system, not only because it guards the original concepts, it actually runs on 68k, because things like necessary functions are still register parametrized, library interface is interchangeable, hook syntax is apparently the same like the original one and last but not least aros has in comparison to the features it offers a reasonable memory footprint, comparable to original kickstarts.

what an irony..

AmigaOS4 still uses fundamentally the same system architecture (warts and everything) which you would find, for example, on page #7 of the 3rd edition RKM "Libraries". The philosophy of keeping the layer between the APIs and the underlying hardware (e.g. Exec, the CPU and memory) thin, so as to deliver the most power to the user, still applies, too.

Beyond that you're bound by the rules imposed by the development of the target hardware platform over the last 15+ years: ABIs, "glue" logic, off-the-shelf components and how the companies which cast all this into a mainboard you can buy and build a system from.

These hardware dependencies, and the new degrees of freedom they afford you are bound to change how the operating system goes about its business. This doesn't fundamentally restrict code written for that platform to be ported. What does make things more complicated is in how the dependencies through APIs evolve.

For example, how the dos.library in AmigaOS4 has evolved is a result of the "traditional" shortcomings of the dos.library design. These being global, public data structures with strange layout, cryptic labeling, simplistic protocols for avoiding clashes between concurrent access, and of course how file systems fit into this picture. It was hard to tell a "hack" from a "best practice" when interacting with dos.library and its data structures.

The dos.library in AmigaOS4 assumes more responsibilities than was the case in previous versions, in which file systems and client software was left to fend for themselves. The old way of doing things required a file system to implement an inane/insane number of operations, accessing crufty data structures according to very underdocumented rules, and this in turn led to stability issues. It was hard to write a robust file system, but poisonously easy to get it subtly wrong with dire consequences, for which it was extremely difficult to figure out what exactly caused them.

So, for example, dos.library now cares about how file change notifications work, how you would change the volume or device list after a medium change. This sounds like the most pedestrian thing to mention, but what goes on under the hood in dos.library V40 (and below) in these areas is a highly complex and error-prone process. Now there are well-documented and well-defined APIs for dealing with such tasks which are easier to use and far less error prone than what was possible before (new APIs and responsibilities, of course, required a more complex implementation, which the more powerful hardware platform certainly enabled).

Consequently, the crufty code which relied on the old methods was replaced, one component at a time, saving space and leading to a more stable and robust system. The Workbench, the shell commands, etc. were modified to use the new APIs.

You could backport such code, but you'd have to drop in replacements for the new API functions found in dos.library. The same relationship exists between the other "pillars" of the Amiga operating system, such as exec.library, graphics.library, intuition.library and the remainder of the operating system.

This is nothing new. It happened during the transition between Kickstart/Workbench 1.3 and 2.04. Even the Commodore developers themselves came to consider the tools (APIs, data structures, etc.) available in the 1.x days as very crude, compared to what 2.04 and beyond would deliver.

Portability of operating system code is not high on the list of things to keep an eye on when creating a new operating system release. If you find that AROS code makes for better backporting, it is likely because the APIs which the code to be ported relies upon, are extremely similar to what existed in 1994. This is not what AmigaOS4 was designed to deliver: it offers API compatibility, extending to data structures in the same way as Kickstart/Workbench 2.04 did to its precursor.

If you've read so far, congratulation! As you might have concluded by now, I do not subscribe to the idea that what makes an Amiga what it is would be strongly connected to how close it is to the operating system design or target hardware of AmigaOS 3.1.

[olsen]

Yeah, like saying AmigaBasic was a Microsoft development.

It wasn't, it was a Metacomco development based on MS source. It's a crying shame how disinformation like that gets put on the internet...

Are you sure?

Metacomco shipped ABasiC (written in 'C') before Microsoft's AmigaBASIC was ready.

As far as I know AmigaBASIC is strongly related to Microsoft BASIC for Mac. As was common at the time, Commodore licensed Microsoft BASIC for the Amiga, with Microsoft creating the Amiga port. Commodore kept paying Microsoft for fixing implementation bugs over the years, but development work ceased by May 1991.

The AmigaBASIC we know from the Kickstart 1.x days had problems on 32 bit systems, e.g. using the 68020 and 68030 CPUs. On the Apple Macintosh it used to be common practice to make use of the restricted address space of the 68000 CPU, with the most significant 8 bits of an address pointer being ignored. Those 8 bits could be used for "pointer tagging", which the 68000 Macintosh operating system employed for memory management uses.

It's possible that the Kickstart 1.x version of AmigaBASIC used the same techniques, or at least assumed that it could tinker with these 8 bits without ill effect. Making AmigaBASIC work robustly on a 32 bit system might have been too expensive for Commodore.

It's also possible that AmigaBASIC was discontinued (I recall Dr. Peter Kittel mentioning that there was a version which ran fine on the Amiga 3000 and worked correctly with Kickstart/Workbench 2.0) because the need for a home computer to ship with BASIC was no longer a given. It certainly was about a decade ago.

[olsen]

No, it is 3.1 allright, and quite a lot more too. Not sure if it also contains things that were worked on under Amiga Technologies, such as stuff for the Walker, but I don't think so, I believe it must be a dump from someone's hard drive at Commodore.

As far as I know the contents of the archive covers the data which all AmigaOS developers could access through the network.

The NFS client software which was part of AS225 was used to connect the Amigas of the developers to the NFS servers which would contain the operating system source code and the material archived/maintained by CATS.

Speculation: I suspect that one of the engineers took a last snapshot of everything before he left Commodore for a related job, e.g. at NewTek, Scala or 3DO. The time stamp of the last file modified is very, very close to the day when Commodore went bancrupt.

[olsen]

Do you have information on how AS225 from CBM became I-Net225 from Interworks?

Part of it may have been licensed/acquired/etc. from Commodore. The authors of "I-Net 225" (Michael B. Smith & Jim Cooper) were part of the networking group at Commodore, who developed the original "A225", "Envoy", "AS225", etc. Commodore (of course!) disbanded the group.

It's possible that both "I-Net 225" and "Envoy" became commercial products because Commodore just didn't see any sense in that Internet thing, local networks, etc.

Commodore certainly lacked the in-house resources to make anything out of the code base, so maybe (I'm speculating) the networking group members were permitted to keep their work and commercialize it.

The "ReadMe" file on the "I-Net 225" installation disk acknowledges the origin of parts of the package.

I quote:

Parts of the "I-Net 225" package were based on a TCP/IP package released by
Commodore Business Machines, Inc.  The original Commodore package (AS225)
was

   Copyright © 1990-1993 Commodore Business Machines, Inc.

End of quote.

I was part of the team which created the "Amiga Surfer" package at Amiga Technologies GmbH back in the winter of 1995/1996. It was built around the Amiga Internet application software available at the time, which included "I-Net 225" and, for example, Oliver Wagner's early "Voyager" browser (bless him). We didn't realize how far ahead of its time this project was when it was under development.

My part of the work concerned the dial-up connection to the Internet, which would use the infrastructure IBM had set up for OS/2. You first had to sign up for an IBM account, and then the access information would be written out as configuration files for "I-Net 225" to use. I wrote the program which sets up the modem, the serial device driver, and which performs the IBM account setup.

Once the configuration files were in place, you could go online by launching the "StartInet" program, if I remember correctly.

And can you confirm that it should be able to connect to Internet (albeit IPv4) using SANA-II ethernet device even today?

Yes. I personally used "I-Net 225" to connect my work machine (an Amiga 3000 UX) to a different Amiga 3000 running NetBSD, with both using "Ariadne" Ethernet cards, back in 1997/1998. I did not connect this setup to the Internet because back then this required a dial-up link, rather than the much more convenient DSL gateway router setups we use today. Had a gateway router been available to connect to the Internet back then, changing the default route to use that router would have easily allowed the "I-Net 225" setup to access the Internet.

The version of "I-Net 225" which I knew did not support DHCP (or BOOTP/RARP), which means that you have to edit/create certain configuration files, knowing exactly which information to enter into them. That's a challenge all by itself.

Setting up the network, however, works differently in "I-Net 225" than in "AmiTCP" or "Miami". Configuration files are stored in "S:", and in "ENV:". The documentation explains how all the pieces need to be set up, it's just that it requires a lot of prior knowledge to put it all together.

In so many words: "I-Net 225" is much harder to set up than the Amiga TCP/IP stacks we use today, but it does get the job done, eventually, if you see it through. "I-Net 225" was the only part of the "Amiga Surfer" package which gave us no trouble at all, and that's saying something.

One more thing: the "I-Net 225" API is different from the one used by "AmiTCP" (and "Miami", "Roadshow", etc.). Back in the early days, application software would support both the "AS225R2"/"I-Net 225" API and the "AmiTCP" API, but nowadays it's unlikely that anything but the "AmiTCP" will be supported. So even if you managed to set up "I-Net 225" correctly, you might be restricted to use the shell commands and applications which ship with "I-Net 225".

[olsen]

How is development of CLI for AROS and MorphOS coordinated with what Thomas here is doing with OS3.x shell?

My guess - not at all

Given that the shell functionality itself is founded on the weird and the wonderful of what the Tripos kernel could provide, I doubt that a coordination would be fruitful. The shell's architecture and its place in AmigaDOS impose constraints which do not (or at least should not) exist in AROS or MorphOS. Not exactly ball and chain, more like aircraft carrier and chain

Now, shell commands, that's a different story, as long as they use public dos.library APIs and data structures.