Wither Natami?

[Piru]

Being one of those developers, and having spent over 7 years on an OS project, I can state with confidence that the_leander got it exactly right. It'd be humongous task.

[Piru]

@Hans_

An updated Amiga OS that breaks compatibility with the old system (but still runs old software in a sand-box) would still have the same design philosophy as the old system. Moreover, you already have a group of like-minded people (which obviously does not include you) ready to make it happen. Start a new OS project, and you end up with "yet another OS project" that is struggling to gain developer interest. Sure it would be a long-term effort, but the starting point is much better than having zero source-code and a blank slate with regard to design.

The problem with this is that if you introduce those new concepts such as MP, SMP, 64bit support and so forth, pretty much all of that old designs and code is useless. Sure you could use those for stuff that runs inside the sandbox, but not for the new stuff.

So basically you are starting with nearly zero source-code and blank slate regarding design. The cynic would ask what exactly would make this new system any better than say Mac OS X or GNU/Linux...

[Piru]

@Hans_

Why?

Say for example everything in AmigaOS and amiga apps depend on the shared memory model.

To mention few inside the OS itself:
- Libraries in general
- Devices (timer, audio, trackdisk, scsi.device etc)
- Resources
- Exec Message passing
- Exec IO interface (devices)
- dos.library <-> filesystem interface
- filesystems (they depend extensively on the message passing)
- Exec interrupt interface
- Hook interfaces all over the place
- ARexx

All of these would need to be rewritten, and since the dependacy on the old legacy APIs would be all over the place, and support for things such as MP, SMP and 64bit sizes would be missing, it would almost be better to write them from scratch anyway.

I also disagree that sandboxing would somehow help migrate the system gradually. You still need something to run the sandbox on. For the sandboxed environment to actually work you'd still need pretty much the whole host opearating system done already. Even with optimistic guestimates that'd take over half a decade (here assuming everything else such as the highly skilled and well motivated  developing team, being paid and doing fulltime work would be provided).

[Piru]

@SamuraiCrow

But such virtual environment needs something to run on top of, too. This is a chicken & egg problem.

[Piru]

@Hans_

Libraries are generally written such that the application holds the data itself.

Unfortunately this is not the case. Most libraries maintain library side data structures which are shared between different callers. These require total rewrite in MP environment.

But how many of the device APIs actually pass pointers to the driver internals? The input device, timer device and others definitely don't. Their message internals could remain the same.

Except that the messaging system must go (see below), which means the device interface will change, too. -> rewriting tons of code.

Exec Message passing

Well, obviously. But what kind of a change are you thinking of? I'd expect the message passing by reference to continue, but with a requirement that the entire contents of a message must be in shared mem (or temporarily shared mem, if possible), and the message receiver isn't allowed to cache pointers (i.e., after a reply message, it no longer has access).

I was merely pointing out the problems, not suggesting any ways to fix them (yes, this is much easier, I know :-)). Since the actual message can contain arbitrary data (say pointers to even more data elsewhere in memory!), it's impossible to re-use this concept in an MP environment. This is one of the largest problems since it means ALL message passing code must be rewritten everywhere. And this problem is extremely widespread since messages are in the core of pretty much every other OS component and application.

Don't IRQ handlers operate in kernel space? If so, what's the issue?

They call user code directly. Assuming each process has it's own memory space this just won't work at all.

How many would involve data crossing over process boundaries? Yes, those would be a real pain.

Most. There are couple of context dependant pointers passed to the hook function for example, and the hooks again call other process' code from the context of either another. Total mess from MP POV. This means hooks must go too, so anything using them must be rewritten.

Exactly what is the difficulty with expanding ints to int64s? We're breaking compatibility anyway, so keeping structure alignment isn't necessary.

Assuming AmigaOS, all of the existing code is hardcoded to 32bit ints (pointer sizes mostly). AROS situation is better here, but AFAIK not everything is fixed yet.

To get things started, you'd need the new exec kernel and a set of device drivers running before the sandbox could be put in place.

Device drivers alone won't do it I'm afraid. Or are you suggesting the sandbox would somehow use the drivers directly? That'd lead to double work since you'd then need to somehow access them from both the sandbox and the new OS itself. Also, not many device drivers handle two simultanous accessors well (for example imagine two OSes fighting over the access to HDD, display framebuffer and input devices). This plan smells of disaster.

One way or another one of the OSes must explicitly manage the actual hardware resources, and then provide them in a safe, controlled manner to the other. If you plan to have MP, this manager can't be the sandbox -> you must have pretty advanced new OS before the sandbox can work.

[Piru]

@Hans_

Such as the library open count? No-one but the system needs access to those.

No. Things inside the library. Things the functions (called from various different processes) modify.

I already said that the pointers to arbitrary memory would no longer be acceptable.

That'd break almost all existing code and you'd need to rewrite much of it. Not to mention there wouldn't be trivial way of detecting wrong usage, other than having the process bomb due to access violation.

the buffers would just have to reside within shared memory

Using shared memory kind of cripples the idea of MP IMHO. Especially since things as DMA can take a very long time and the buffer would sit exposed for extended periods.

Also, how is this shared memory going to work? By forcing each memory allocation to pagesize alignment? Or by copying the whole message around at putmsg and replymsg? Using copy sounds pretty bad for anything that passes largish data around. Forcing alignment to pagesize doesn't sound particularily appealing either.

typedef void* APTR;

Except that there are million things that don't use it.

Doing a search and replace for LONG would be a good start (just not through all source-files in one shot).

Manually check every occurance of LONG? Sounds pretty mind numbing, indeed...

The sand-box would be more for running the existing GUI system and other higher-level stuff. Basically anything that's not exec, DOS or a device.

So to recap: You must write much of the new system before you actually get to run the sandbox.

[Piru]

@Hans_

Could you please provide a solid example so that I understand what the problem is?

selib.c

Constructs such as this are extremely common, and WILL die horribly with MP.

I don't think that a manual check would be needed for every occurrence.

Why not?

[Piru]

@HenryCase

1. Will copying or adapting some of the design features from CAOS allow us to move into a more stable AmigaOS without having to rewrite all the subsystems from scratch?

No.

[Piru]

@SamuraiCrow

Hosted AROS runs AROS software in a sandbox on top of Linux. If you don't want the sandbox and want to run it memory-protected, use Linux itself.

Examundo. You hit the proverbial nail in the proverbial head.

This is the exact reason why such NG-AmigaOS-project makes absolutely no sense at all. You'd only end up with 0 applications, especially if you wouldn't be POSIXy (apparently some see it as a bad thing...). Also, lets not forget the horrible pain that'd ensue from having to write drivers for the tens of thousands of x86 devices. Either do that, or severely limit the possible setups for the OS.

I'd take the negligible overhead of running hosted on top of the Linux kernel any day.

[Piru]

@Hans_

Looking at it quickly, you have a global list containing nodes from every library user.

Not quite. There can be any number of them per caller.

One option would be to make the list a per-library open item. This would require the addition of a single keyword.

How would that fix it exactly?

I still think that the bulk of most library source-code wouldn't be affected.

Well I disagree. This is just one example of how resources allocated by any function can totally screw it up in MP environment. Many libraries allocate resources in the functions that are somehow referenced by the other library openers.

Because everything that fits into an int32 fits into an int64. Specific sizings usually relate to hardware interfacing and really low-level stuff.

That'd break many more things than you think, for example files read from/written to disk, network or any other medium. You can't just do global search and replace like that, it'd b0rk horribly. Another suckage from this would be the needless extra memory consumption. It'd also suck to have every other 32bits "unused", esp for caches.

[Piru]

@HenryCase

If you think about it for a second you realize CAOS is totally different from current amigaos. Thus, if you'd replace parts of the amigaos with CAOS, nothing would work anymore. Thus, No, it would not help at all.

How hard is it to port a POSIX compatible app to a non-POSIX OS, in terms of libraries missing and other needed infrastructure?

Depending on which missing functions are used, pretty hard to totally impossible.

{EDIT] Well, nothing is impossible, but if you need to rewrite most of the code then I'd say "porting" didn't happen :-) [/EDIT]

[Piru]

@Hans_

I'm assuming that you're not trying to use the library for inter-task communications here.

I am not. Lets just assume the library needs to access the other nodes for some reason. Even though you seem to have missed such constructs, they are common in real life, indeed. My example has just some dummy code that illustrates the problem.

And how many of those would actually require the other tasks to reference resources not allocated by them?

I can't see how it could be avoided really. These libraries really need to access the structures allocated by other callers.

[Piru]

:-)

[Piru]

You said that basically the whole shared library method would have to be discarded due to memory sharing issues.

And my example only provided one reason for it.

Others include the problem the fact that you'd need to have different library base per caller. Something that doesn't work with many libraries.

I need a solid example why a library would need to allow multiple callers to access structures allocated by the other.

diskfont.library, dos.library, asl.library, locale.library, rexxsyslib.library, realtime.library, commodities.library, datatypes.library, card.resource, timer.device, keyboard.device, trackdisk.device, graphics.library, intuition.library, layers.library, gadtools.library, exec.library, utility.library at least.

Of 3rd party libs reqtools, xadmaster, xfdmaster, openurl, muimaster, ixemul at least.

[Piru]

@Hans_

Naming libraries is not a solid example.

So if I explain why one of the libraries requires it will that be enough for you? (The same problem is in all of the libraries I listed. It is the very same problem I tried to demonstrate with the example library code, but apparently failed miserably)