Wither Natami?

[Hans_]

Ok, but how would breaking compatibility with your current software (as you have conceded as a necessity toward making the OS serviceable in a modern context) going to help you over porting it to such a hybrid, or getting it running under UAE?

Having a similar API would ease the porting process.

Further, If you find UAE slow, try Amithlon, seriously, the difference is night and day and even on a system such as a 901 or 1001 EeePC, likely faster and more responsive then any Amiga or AmigaOne ever built.

It's no longer available. Amithlon could have become the basis for a new Amiga OS. Too late now with all the developments that have happened since.

Hans

[Hans_]

@Piru

Piru wrote:
@Hans_

Libraries are generally written such that the application holds the data itself.

Unfortunately this is not the case. Most libraries maintain library side data structures which are shared between different callers. These require total rewrite in MP environment.

Such as the library open count? No-one but the system needs access to those.

Exec Message passing

Well, obviously. But what kind of a change are you thinking of? I'd expect the message passing by reference to continue, but with a requirement that the entire contents of a message must be in shared mem (or temporarily shared mem, if possible), and the message receiver isn't allowed to cache pointers (i.e., after a reply message, it no longer has access).

I was merely pointing out the problems, not suggesting any ways to fix them (yes, this is much easier, I know :-)). Since the actual message can contain arbitrary data (say pointers to even more data elsewhere in memory!), it's impossible to re-use this concept in an MP environment. This is one of the largest problems since it means ALL message passing code must be rewritten everywhere. And this problem is extremely widespread since messages are in the core of pretty much every other OS component and application.

I already said that the pointers to arbitrary memory would no longer be acceptable. I even discuss pointers in the bit that you quoted. Now why would the timer messages have to be changed? I think that the driver messages with pointers to data buffers could remain mostly intact too; the buffers would just have to reside within shared memory, or within the message allocation itself. That's going to annoy some app developers, but I don't see that as a reason to completely dump the message passing scheme.

Exactly what is the difficulty with expanding ints to int64s? We're breaking compatibility anyway, so keeping structure alignment isn't necessary.

Assuming AmigaOS, all of the existing code is hardcoded to 32bit ints (pointer sizes mostly). AROS situation is better here, but AFAIK not everything is fixed yet.

typedef void* APTR;
Doing a search and replace for LONG would be a good start (just not through all source-files in one shot). This would be more of a tedious task than a hard one.

To get things started, you'd need the new exec kernel and a set of device drivers running before the sandbox could be put in place.

Device drivers alone won't do it I'm afraid. Or are you suggesting the sandbox would somehow use the drivers directly? That'd lead to double work since you'd then need to somehow access them from both the sandbox and the new OS itself. Also, not many device drivers handle two simultanous accessors well (for example imagine two OSes fighting over the access to HDD, display framebuffer and input devices). This plan smells of disaster.

Anything in the sandbox would have to go through an abstraction layer, so you'd likely have to write a scsi.device, trackdisk.device, etc. that would go through the abstraction layer (yes, more work I know).

The sand-box would be more for running the existing GUI system and other higher-level stuff. Basically anything that's not exec, DOS or a device.

Having said that, some drivers could initially be allowed to access hardware "directly" (still going through some kind of layer) from the sand-box until a replacement exists for the new OS. The idea is that, initially, the new OS would only be running one app, the sand-box. So, having something like the USB stack running in the sand-box and providing it with exclusive access to the USB card wouldn't be a big deal as the new OS wouldn't have its own USB stack yet. As soon as the new OS gets its own stack, the sand-box would require a wrapper.

Hans

[Hans_]

the_leander wrote:

Hans_ wrote:

Ok, but how would breaking compatibility with your current software (as you have conceded as a necessity toward making the OS serviceable in a modern context) going to help you over porting it to such a hybrid, or getting it running under UAE?

Having a similar API would ease the porting process.

But no moreso then using the hypothetical Haiku/AROS hybrid from what I can see.

I am assuming that the message passing system will still be similar, instead of the complete replacement that Piru says would be required. That would make it easier.

Hans

[Hans_]

Piru wrote:
@Hans_

Such as the library open count? No-one but the system needs access to those.

No. Things inside the library. Things the functions (called from various different processes) modify.

Could you please provide a solid example so that I understand what the problem is?

I already said that the pointers to arbitrary memory would no longer be acceptable.

That'd break almost all existing code and you'd need to rewrite much of it.

Inserting appropriate message allocations would be tedious, but it's not like you have to write things from scratch.

Not to mention there wouldn't be trivial way of detecting wrong usage, other than having the process bomb due to access violation.

Now this is something that I just didn't think about. It would be nasty if you could crash a process just by sending it a bad message.

the buffers would just have to reside within shared memory

Using shared memory kind of cripples the idea of MP IMHO. Especially since things as DMA can take a very long time and the buffer would sit exposed for extended periods.

The memory wouldn't have to be shared with everyone. Nor would it have to be writable by all processes with which it's shared.

Also, how is this shared memory going to work? By forcing each memory allocation to pagesize alignment? Or by copying the whole message around at putmsg and replymsg? Using copy sounds pretty bad for anything that passes largish data around. Forcing alignment to pagesize doesn't sound particularily appealing either.

Large messages (i.e., ones with big buffers) could be page-sized. For small messages a copy would probably be the sensible thing to do.

typedef void* APTR;

Except that there are million things that don't use it.

Hence the next line below.

Doing a search and replace for LONG would be a good start (just not through all source-files in one shot).

Manually check every occurance of LONG? Sounds pretty mind numbing, indeed...

I don't think that a manual check would be needed for every occurrence.

The sand-box would be more for running the existing GUI system and other higher-level stuff. Basically anything that's not exec, DOS or a device.

So to recap: You must write much of the new system before you actually get to run the sandbox.

Correct. Although as I said, initially some of the drivers could be inside the sand-box with exclusive access to the hardware. That would be the case until a native version was available for the new system.

Hans

EDIT: I've just realized how much time I've wasted debating something that won't happen any time soon (if ever).

[Hans_]

bloodline wrote:

Hans... no offence meant here, but have you ever done any Amiga development at all? I ask because the questions you are asking show a real lack of knowledge with AmigaOS... If you want to find out more about how AmigaOS works, just have a look over the AROS source code it would give you a much clearer idea of the scale of the problem!

How do the questions that I'm asking give the impression that I don't know anything about Amiga OS? I don't see how discussing some of the specific internals demonstrates a complete lack of understanding ... no offence meant of-course. ;-)

FYI, yes I have done Amiga development, and I do know a thing or two about how Amiga OS works. Obviously my knowledge is not as extensive as someone that has actually worked on the OS itself, but I do understand the message passing system, devices, libraries, etc.

Hans

[Hans_]

HenryCase wrote:

bloodline wrote:
That was a legitimate question

Maybe, but what did you intend to achieve by asking it?

I don't see it as a legitimate or useful question actually. I find it hard to see how what I wrote demonstrates a complete lack of understanding. The question serves only to question my intelligence and understanding and does nothing to debate the actual issues. It's just part of the negative rhetoric along with comments such as "... silly pipe-dreams ..." Bloodline often has a mocking tone in his posts which you'll just have to get used to.

bloodline wrote:
What "OpenGL stuff"?

Adding functions/features to TinyGL.

Correction, I'm extending MiniGL. I have worked on other stuff, but none of that is released yet (and some of it is discontinued).

bloodline wrote:
if you read what is left of the specs, you can see it was an odd system... probably even less compatible with the all pervasive POSIX than what we have now...

Not everyone sees POSIX as a good thing. Seems to me the more you strive for total POSIX compatibility the more you turn your OS into a UNIX variant. As long as apps aren't super difficult to port why care about POSIX?

Like it or not POSIX is now an official standard and a lot of code has been written that uses it. Any new system that ignores POSIX is doomed.

Hans

[Hans_]

Piru wrote:
@Hans_

Could you please provide a solid example so that I understand what the problem is?

selib.c

Constructs such as this are extremely common, and WILL die horribly with MP.

Thanks for the example.

Looking at it quickly, you have a global list containing nodes from every library user. One option would be to make the list a per-library open item. This would require the addition of a single keyword. Another option would be to leave it in shared memory, which you obviously wouldn't like, and probably wouldn't be necessary. I still think that the bulk of most library source-code wouldn't be affected.

The libraries that I can think of that really do require common internal state variables would be things such as the bsdsocket library which is part of the TCP/IP networking stack. An MP environment would require the library to pass messages to the TCP/IP process for almost everything, if this isn't how it's done already.

I don't think that a manual check would be needed for every occurrence.

Why not?

Because everything that fits into an int32 fits into an int64. Specific sizings usually relate to hardware interfacing and really low-level stuff. Also note that a 64-bit version of AROS already exists.

[Hans_]

Piru wrote:
@Hans_

Looking at it quickly, you have a global list containing nodes from every library user.

Not quite. There can be any number of them per caller.

Containing nodes from every library user includes multiple nodes per caller.

One option would be to make the list a per-library open item. This would require the addition of a single keyword.

How would that fix it exactly?

It would be a local list sitting within the caller's memory space. I'm assuming that you're not trying to use the library for inter-task communications here.

I still think that the bulk of most library source-code wouldn't be affected.

Well I disagree. This is just one example of how resources allocated by any function can totally screw it up in MP environment. Many libraries allocate resources in the functions that are somehow referenced by the other library openers.

And how many of those would actually require the other tasks to reference resources not allocated by them?

Because everything that fits into an int32 fits into an int64. Specific sizings usually relate to hardware interfacing and really low-level stuff.

That'd break many more things than you think, for example files read from/written to disk, network or any other medium. You can't just do global search and replace like that, it'd b0rk horribly. Another suckage from this would be the needless extra memory consumption. It'd also suck to have every other 32bits "unused", esp for caches.

Fair enough. Note, I did not suggest just running a search-replace on the entire source tree.

Hans

[Hans_]

Hans_ wrote:
Like it or not POSIX is now an official standard and a lot of code has been written that uses it. Any new system that ignores POSIX is doomed.

I'd still personally resist a 'one size fits all' computing world even if there were significant drawbacks. Got to give the world a true alternative to POSIX one day IMO.

There are alternatives, but almost no-one uses them. Developers like standards because it allows them to reuse code on multiple systems. This is why everyone uses the SCSI or IDE standards for drive interfacing; why ATAPI for IDE is based on the SCSI protocol; why SATA is based on ATA, etc. It's also why Warp3D Nova won't be created; instead OpenGL will be the 3D API.

I'd be fine with ignoring POSIX compliance if there were some clear advantages to it. For desktop computing, there are none (that I can think of). Besides, POSIX compliance can mean that there is simply a POSIX API wrapper sitting on top of the OS' API. That's the way that POSIX thread works on Amiga OS for example (and Windows).

Windows is fully POSIX compliant, MacOSX is fully POSIX compliant, as are a whole range of other OSes. Surprizingly, Linux is listed as being mostly POSIX compliant (not fully).

Hans

[Hans_]

Piru wrote:
@Hans_

I'm assuming that you're not trying to use the library for inter-task communications here.

I am not. Lets just assume the library needs to access the other nodes for some reason. Even though you seem to have missed such constructs, they are common in real life, indeed. My example has just some dummy code that illustrates the problem.

I see no problem in your example code. You said that basically the whole shared library method would have to be discarded due to memory sharing issues. I see nothing in your example code that supports that.

And how many of those would actually require the other tasks to reference resources not allocated by them?

I can't see how it could be avoided really. These libraries really need to access the structures allocated by other callers.[/quote]
You didn't answer the question, and simply saying that it is necessary doesn't help. I need a solid example why a library would need to allow multiple callers to access structures allocated by the other.

Hans

[Hans_]

Piru wrote:

You said that basically the whole shared library method would have to be discarded due to memory sharing issues.

And my example only provided one reason for it.

No it didn't. I do not see any problem in your example that would kill the shared library model.

I need a solid example why a library would need to allow multiple callers to access structures allocated by the other.

diskfont.library, dos.library, asl.library, locale.library, rexxsyslib.library, realtime.library and commodities.library at least.

Of 3rd party libs reqtools, xadmaster, xfdmaster, openurl, muimaster at least.

Sigh. Naming libraries is not a solid example.

Hans

[Hans_]

Piru wrote:
@Hans_

Naming libraries is not a solid example.

So if I explain why one of the libraries requires it will that be enough for you?

Yes. Assuming that I can't think of a method of dealing with the problem.

Hans

[Hans_]

the_leander wrote:

Hans_ wrote:

I need a solid example why a library would need to allow multiple callers to access structures allocated by the other.

diskfont.library, dos.library, asl.library, locale.library, rexxsyslib.library, realtime.library and commodities.library at least.

Of 3rd party libs reqtools, xadmaster, xfdmaster, openurl, muimaster at least.

Sigh. Naming libraries is not a solid example.

Hans

Perhaps you might consider Bloodline's suggestion to download the AROS source? Maybe even look at the aros forums to see where all the problems are.

Searching for a problem that you think doesn't exist is kind of hard.

Hans

[Hans_]

Piru wrote:
@Hans_

utility.library/AddNamedObject
utility.library/RemNamedObject
utility.library/AttemptRemNamedObject

These named objects are in a global tree which can be modified by any caller. For additional headache, the objects can be removed by different task, too. Read the autodocs for details.

Ok, an API of a library that needs to be changed. If the named objects were used within a single application then it wouldn't be a problem with a threading model (i.e., extend the idea of a sub-process or thread to the memory protection scheme). It shouldn't be used to communicate/transfer objects between applications.

dos.library/AddDosEntry

This function adds the given dosentry to the internal global doslist. Some things (filesystems, apps) allocate the dosnodes themselves and pass them to the AddDosEntry (among other things it means you can't safely free a dosentry you remove). With MP this concept would be horribly broken.

Again, library specific API design. We've already talked about how the DOS library would need a lot of work.

ramlib
exec/OpenLibrary
exec/OpenDevice
exec/OpenResource
exec/AllocMem etc

ramlib calls LoadSeg to load the library/device/resource, yet expunge gets called by totally unrelated task:

If system runs out of memory or "avail flush" is performed, the system will attempt to free memory by calling the "lowmemhandlers". These handlers get called in the context of the AllocMem caller, meaning that different task will call library/device/resource Init and Expunge vectors.

In an MP environment, the library/device/resource code would have to be loaded into memory that is marked as shared read-only and executable. As such the whole loading and unloading mechanism would have to be modified to make that happen. Just because everything currently happens in the caller's task doesn't mean that it has to be that way. An application programmer won't notice the difference if you change the way that libraries load.

You've identified a number of issues with existing APIs that would indeed have to be addressed, but nothing that I would say kills the shared library system. Really, if shared objects/DLLs work on MP systems then I see no reason why shared libraries would fail.

Hans

[Hans_]

Piru wrote:

You've identified a number of issues with existing APIs that would indeed have to be addressed, but nothing that I would say kills the shared library system.

The MP issues I've listed are just part of the problem, more like annoyances, icky things you would need to fix all over the place. There are other similar issues that add to the workload, such as global variables being used on disk based libraries.

Each caller would need to get their own library base. Also, to allow static data (disk based libraries) you'd need to somehow clone the data hunks for each caller and handle the relocation. In all this would end up being closer to .so than the classical shared library system. Thus my claim that it would kill the shared library system (as we know it).

There are already libraries that provide each caller with their own base. Static data could be left in read-only shared memory. It is true that shared libraries would become even more like shared objects.

So the only bit that we're really disagreeing on is what constitutes "killing" the shared library system. Well that took a lot of posts to figure out.  :lol:

So, to summarize a bit:
- Piru: I spent over seven years doing that kind of ____ and I certainly don't want to do it all again from scratch. Look at all the stuff you'd have to do.
- Hans: It's a big task, but not too big. Plus I'm a glutton for punishment. Let's make it happen.

Hans

[Hans_]

kolla wrote:

Windows is fully POSIX compliant, MacOSX is fully POSIX compliant, as are a whole range of other OSes. Surprizingly, Linux is listed as being mostly POSIX compliant (not fully).

Hans

This must be the first time I have ever seen anyone claim Windows to be fully POSIX compliant. What do you mean, through cygwin? I think you're rambling.

How about you ramble over to the wikipedia page for POSIX. It used to state full compliance, now it lists the following for windows:

   * Cygwin – enables partial POSIX compliance for certain Microsoft Windows products.
    * Microsoft Windows Services for UNIX 3.5 – enables full POSIX compliance for certain Microsoft Windows products.
    * UWIN from the AT&T Research implements a POSIX layer on top of the Win32 APIs.

So, you need the business version of Windows for full POSIX compliance. Vista has been advertised as being able to run Unix software. Cygwin (and probably UWin) are partial POSIX layers.

So, let me revise that. Some versions of Windows are fully POSIX compliant.

Hans