How to move AROS forward

[Piru]

@HenryCase

let's start discussing how to graft full MP into AROS through program compilers.

MP is not the task of the compiler.

Would there be any way to make AllocMem (and all other AROS functions) behave as they should without using pointers (maybe using some new 'port' system instead that AROS could flag and control)?

Even if you could, the actual poking can be in the code, not the OS itself. The code needs to be rewritten, compiler can't possibly know how to do that for you.

[Trev]

My two cents re: interpretation of binary data: It's possible to interleave opcodes and operands, where one opcode's operand is actually an opcode itself, and a subsequent jump or branch takes advantage of that fact. This was a common dirty hack in MS-DOS software. You can't assume that every byte (or every bit, even) is either an opcode or an operand. It can be both. Or padding. Or a gamma ray induced parity error. Or an abstract picture of someone's grandmother.

Obviously, that complicates any attempt to map an arbitrarily complex sequence of code. If you do figure out a way to do it optimally in polynomial time, kudos. You'll earn a Fields Medal (if you're under 40 years of age), and the Clay Institute will award you 1,000,000 USD for a generalized solution. Hell, you'd probably earn a Nobel Prize as well. ;-) For all we know, you could be the one who solves this problem. Good luck! Sincerely.

Trev

[Belial6]

now this assembles to:
0x10011101120313

that's our actual binary executable for our mythical computer there.

Now, suppose I write a program (for a version of this computer with more memory and some I/O capability) which stores a user password. This will be done by using two four-bit 'bytes' per character, so we'll use ASCII. Only we're going to use ROT-n to encode the passwords so they can't be as easily read. Arbitrarily, the program has selected -21 for the offset, n, so each letter has 21 subtracted from it's ASCII value before storage. So if the user enters password "#l|l}n~" we get 0x10011101120313 as the encoded password.

That would be a cool 'Demo' challenge.  Make programs that are actual sentences in ascii, but do when run as code, do something... or nothing, but don't crash.

[bloodline]

HenryCase wrote:

Bingo. CPU emulation is part of my plan. Yes it is too slow to use for running programs in real time, but it can be used to construct a 'map' of the program. This 'map' would only need to be created once, but would then be used by the operating system to help guide the program execution at key points.

If you could build a map, you wouldn't need Memory Protection. If you can build a map then you already know every possible outcome of running the program. If fact you wouldn't ever need to run the program, instead you would just look at the end of the map and see the outcome of the execution! And when you see it like that you realise that the map would be gigabytes in size for even the simples program.

[bloodline]

HenryCase wrote:
@Piru
As I said before, I am currently looking of ways around this issue.

First you need to understand the problem... at the moment you are looking at the symptoms of the problem and trying to make the symptoms go away, without actually sorting out the underlying problem.

The code map may end up being in a more high level modular structure.

A map is a map... I don't think you understand the scale of what you are suggesting!

Maybe also we keep a record of the code paths we have taken before, and if a new one arises we track its performance, adding to our code map.

Imagine you are trying to plot the Code path for a game, lets take Quake... you would have to make a code path for every single possible variation in movement made by the player, at every frame of the game... I don't think that you could map that in a thousand years...

How do you deal with prorgams that are entirely intimate... one map for both programs? one map for each program, but which have common parts? How do you deal with the fact that in AmigaOS, everything essentially runs as one big program?

These ideas are only rough sketches for now, I only realised through this thread that I'd have to deal with pointer issues when retrofitting Amiga operating systems and apps with MP, so forgive me for not going into more detail.

A Sketch it may be... but already you should be able to see that it's a flawed idea, and one that actually does nothing to solve our initial problem... which is that the OS has no way to know what a program is up to and hence no way to enforce any restrictions upon those programs.

@all
Whilst I carry on sketching out how to track the performance of Amiga programs, I'd like to take the discussion about MP on a simpler route. Seeing as the whole idea of AROS being OS 3.1 compatible is on a source code level (apart from the plans for binary compatibility on the 68k port), let's start discussing how to graft full MP into AROS through program compilers.

This doesn't make any sense. A compiler cannot add memory protection to a program... Just as a dictionary does not make for good bed time reading.

Would there be any way to make AllocMem (and all other AROS functions) behave as they should without using pointers (maybe using some new 'port' system instead that AROS could flag and control)?

Do you even know what a pointer is? How can AllocMem() work without returing a pointer to the memory you have just allocated? Could you write a story without using verbs?

Have you googled MicroSoft's Singularity yet? I think there is an OS much better suited to your needs than Amiga OS ever will be...

[HenryCase]

@all
I'm at work so I don't have time to reply to all of your points.

@Trev

Trev wrote:
the Clay Institute will award you 1,000,000 USD for a generalized solution.

You get 1 million USD for solving that 400 students problem?! I've already started. ;-)

Piru wrote:
The code needs to be rewritten, compiler can't possibly know how to do that for you.

Yes it can. Let's say you rewrite the compiler to add a "-MPAROS" compiler flag. Then whenever you try and compile programs with this flag it follows certain rules to make programs compatible with an MP kernel. The resultant code is different from compiling it normally, but crucially you wouldn't need to rewrite the application code yourself.

[bloodline]

But henry, you are missing the point. The OS already knows when a message is sent. The OS has quite a bit of information regarding message actually! But the OS does not know what the contents of the message is or how it is to be used... The compiler doesn't know this either.

The problem comes when that message contains references to memory areas not in the context of the receiving task... In AmigaOS this is fine... In an MP OS this is bad.

And right now we are only talking about Data... What if one task jumps to some code that is part of another task's context? BANG... One dead task... And exec is built around this concept!

[Hans_]

HenryCase wrote:
Yes it can. Let's say you rewrite the compiler to add a "-MPAROS" compiler flag. Then whenever you try and compile programs with this flag it follows certain rules to make programs compatible with an MP kernel. The resultant code is different from compiling it normally, but crucially you wouldn't need to rewrite the application code yourself.

Let me give you an idea as to how difficult this would be. An MP version of the Amiga OS API would have to place restrictions on memory used in messages. All memory referenced in a message would have to be shared, or at least shared with the recipient. Old programs could use any memory, and might even allocate a message on the stack (which would be private). So the first step would be to re-allocate the message in shared memory.

So far so good. Unfortunately, this message could contain pointers to other memory. A compiler might know that some are pointers. However, coders have a tendency to do naughty things, and might write a pointer into a variable declared as an integer. Even if a coder does follow the rules, a pointer to an array or image, won't tell you how big that array or image is; and, if it's a dynamically allocated memory, you won't know at compile time (i.e., the compiler won't be able to do anything).

Now, to top everything off, this whole thing is recursive. The memory that is pointed to could point to other memory, which would point to other memory, etc.

This is why old apps would have to run in a sand-boxed environment. Old apps would collectively run in a shared memory space (so yes, they can interfere with one-another), and any interaction with the OS would have to go through a compatibility layer that would repackage calls to use the new API.

Hans
 

[koaftder]

@HenryCase

I wrote a small, very simple virtual machine last weekend which you may or may not find interesting along with a simple assembler. At the very least it shows how a simple vm can be written. It's a stack machine with ~25 opcodes and no registers.

The VM

assembler

example assembly program

It does have bugs.

[HenryCase]

@bloodline and @Hans_
Thank you both again for informative posts. I think you misunderstand my plan, but that's understandable as I haven't explained it fully. Think of it like API translation at the compiler level. So instead of declaring AllocMem() as a pointer, the same attributes about the memory requirements (memory size, flags) get declared in a malloc kind of way. What I aim to do is take the API calls and turn them into ones that can support MP. Does this sound more reasonable?

@koaftder
Cool. I'll check out your VM when I get time ("99 bottles of beer on the wall, 99 bottles of beer...").

[bloodline]

HenryCase wrote:
@bloodline and @Hans_
Thank you both again for informative posts. I think you misunderstand my plan, but that's understandable as I haven't explained it fully.

I understand your plan perfectly!

Think of it like API translation at the compiler level. So instead of declaring AllocMem() as a pointer, the same attributes about the memory requirements (memory size, flags) get declared in a malloc kind of way. What I aim to do is take the API calls and turn them into ones that can support MP. Does this sound more reasonable?

What you haven't realised is that the function call (in this case AllocMem() ) isn't the problem. It is just an interface, the problem is how one is allowed use the API. That was defined by the behaviour of the OS and apps, in such a way as to prevent the ability to retrofit Memory Protection.


In an MP OS, one would define that the pointer returned by AllocMem() is for the use of the calling task only.
In AmigaOS the pointer returned by AllocMem() is free for any task to use, this is defined behaviour.

If you change the above definition you no longer have an AmigaOS compatible system, and everything would have to be rewritten, all the various subsystems would need to be altered in the way they work to the point that no programer would recognise the system as AmigaOS... It would probably look much more like Linux. And no Amiga Programs would compile for it anymore either...

The next problem we have is that function calls in AmigaOS are simple jumps from one executable to another... in an MP OS you have to trigger an interupt, so the machine can enter supervior mode, and then deal with the data as required.

@koaftder
Cool. I'll check out your VM when I get time ("99 bottles of beer on the wall, 99 bottles of beer...").

First I tihnk you should get to grips with how AmigaOS works... get a pen and paper and plot the interactions, once you see it written down the problems will be plain to see!

[Hans_]

@HenryCase

HenryCase wrote:
@bloodline and @Hans_
Thank you both again for informative posts. I think you misunderstand my plan, but that's understandable as I haven't explained it fully. Think of it like API translation at the compiler level. So instead of declaring AllocMem() as a pointer, the same attributes about the memory requirements (memory size, flags) get declared in a malloc kind of way. What I aim to do is take the API calls and turn them into ones that can support MP. Does this sound more reasonable?

The problem is that not all messages are allocated using AllocMem() (people should be using AllocVec()/AllocVecPooled() anyway). Messages that were allocated on the stack would not work with such a scheme. You are still going to encounter every problem that we've outlined. Sooner or later you are going to encounter things that cannot be determined at compile-time.

In theory it's all possible simply because we know that humans can do it. However, you will rapidly discover that human thought-processes are way more complex than anything that you can program. Moreover, it could take a human programmer months to make the transition, including extensive testing. Expecting a compiler to do this in seconds is unrealistic.

All of what I've written above ignores the fact that a developer would structure message contents differently in a full MP environment. Pointers to structures would be used less, strings would probably be dynamically allocated instead of just using a pointer to a constant, etc.

What exactly is wrong with allowing old software to be fully-allocated in shared-memory and run in a sand-box?

Hans

[Hans_]

bloodline wrote:

In an MP OS, one would define that the pointer returned by AllocMem() is for the use of the calling task only.
In AmigaOS the pointer returned by AllocMem() is free for any task to use, this is defined behaviour.

If you change the above definition you no longer have an AmigaOS compatible system, and everything would have to be rewritten, all the various subsystems would need to be altered in the way they work to the point that no programer would recognise the system as AmigaOS... It would probably look much more like Linux. And no Amiga Programs would compile for it anymore either...

It's not quite that bad. Amiga OS 4 programs can request private memory with the down-side that it's per task. There is no concept of multiple threads inside a process (or sub-processes, or whatever), so you can't perform direct sharing within a multi-threaded program. In a full MP OS, memory would be private by default, and you'd have to explicitly ask for shared memory. Henry is suggesting getting the compiler to insert the MEMF_SHARED flag (and/or whatever else woulld be used) whenever it's necessary. The problem is that finding out when it's necessary is incredibly complex.

There is no reason to completely drop message passing by reference. The only requirement would be that the message doesn't contain pointers to private memory.

The next problem we have is that function calls in AmigaOS are simple jumps from one executable to another... in an MP OS you have to trigger an interupt, so the machine can enter supervior mode, and then deal with the data as required.

Ok, now you've lost me. On all OSes, function calls to libraries are simple jumps. Sending messages between processes require an interrupt. However, that's exactly the same in Amiga OS. Amiga OS programs don't just suddenly start executing each other's code within their own task/process. If a message is sent, one task/process is put to sleep (at the interrupt), and the other is woken up by the OS so that it can respond.

Hans

[bloodline]

Hans_ wrote:

bloodline wrote:

In an MP OS, one would define that the pointer returned by AllocMem() is for the use of the calling task only.
In AmigaOS the pointer returned by AllocMem() is free for any task to use, this is defined behaviour.

If you change the above definition you no longer have an AmigaOS compatible system, and everything would have to be rewritten, all the various subsystems would need to be altered in the way they work to the point that no programer would recognise the system as AmigaOS... It would probably look much more like Linux. And no Amiga Programs would compile for it anymore either...

It's not quite that bad. Amiga OS 4 programs can request private memory with the down-side that it's per task. There is no concept of multiple threads inside a process (or sub-processes, or whatever), so you can't perform direct sharing within a multi-threaded program. In a full MP OS, memory would be private by default, and you'd have to explicitly ask for shared memory. Henry is suggesting getting the compiler to insert the MEMF_SHARED flag (and/or whatever else woulld be used) whenever it's necessary. The problem is that finding out when it's necessary is incredibly complex.

There is no reason to completely drop message passing by reference. The only requirement would be that the message doesn't contain pointers to private memory.

Hans, please don't complicate things! I'm well aware of more advanced strategies used to improve performance, but Henry is having a hard enough time grasping the basics here. He needs to understand that the messages and message passing are not the problem, it's the contents that's the problem.

So I would prefer it if we could keep a examples as simple as possible.

The next problem we have is that function calls in AmigaOS are simple jumps from one executable to another... in an MP OS you have to trigger an interupt, so the machine can enter supervior mode, and then deal with the data as required.

Ok, now you've lost me. On all OSes, function calls to libraries are simple jumps.

I'm not talking about libraries, I'm talking about how you make a system cal to the Kernel. On AmigaOS Kernel calls are simple jumps the same as a library, but on a Real OS, that has a separate kernel space, you then need to use an interrupt to jump into Kernel space...

Sending messages between processes require an interrupt. However, that's exactly the same in Amiga OS. Amiga OS programs don't just suddenly start executing each other's code within their own task/process. If a message is sent, one task/process is put to sleep (at the interrupt), and the other is woken up by the OS so that it can respond.

But calling the message function does not trigger a reschedule, AFAIR... thus does not interrupt...

Anyway, my point was to show how separate address spaces actually work... Henry is missing a fundamental point of MP. It is this point we need to address.

[Hans_]

bloodline wrote:

Ok, now you've lost me. On all OSes, function calls to libraries are simple jumps.

I'm not talking about libraries, I'm talking about how you make a system cal to the Kernel. On AmigaOS Kernel calls are simple jumps the same as a library, but on a Real OS, that has a separate kernel space, you then need to use an interrupt to jump into Kernel space...

Ok.

Sending messages between processes require an interrupt. However, that's exactly the same in Amiga OS. Amiga OS programs don't just suddenly start executing each other's code within their own task/process. If a message is sent, one task/process is put to sleep (at the interrupt), and the other is woken up by the OS so that it can respond.

But calling the message function does not trigger a reschedule, AFAIR... thus does not interrupt...

The other task won't receive the message until a context switch (requiring an interrupt) occurs.

Anyway, my point was to show how separate address spaces actually work... Henry is missing a fundamental point of MP. It is this point we need to address.

There's no need for separate address-spaces. Full memory-protection can be achieved without it.

Hans

[bloodline]

Hans_ wrote:

bloodline wrote:

Ok, now you've lost me. On all OSes, function calls to libraries are simple jumps.

I'm not talking about libraries, I'm talking about how you make a system cal to the Kernel. On AmigaOS Kernel calls are simple jumps the same as a library, but on a Real OS, that has a separate kernel space, you then need to use an interrupt to jump into Kernel space...

Ok.

Sending messages between processes require an interrupt. However, that's exactly the same in Amiga OS. Amiga OS programs don't just suddenly start executing each other's code within their own task/process. If a message is sent, one task/process is put to sleep (at the interrupt), and the other is woken up by the OS so that it can respond.

But calling the message function does not trigger a reschedule, AFAIR... thus does not interrupt...

The other task won't receive the message until a context switch (requiring an interrupt) occurs.

Of course :-) But the actual message passing does not trigger one.. I could be wrong, here it's not really very important, we are just arguing semantics, so hopefully Henry will ignore this part of the thread.. I'll check the RKMs later and confirm.

Anyway, my point was to show how separate address spaces actually work... Henry is missing a fundamental point of MP. It is this point we need to address.

There's no need for separate address-spaces. Full memory-protection can be achieved without it.

Lets not confuse Henry anymore, please :-) We need to present the simplest example, since that gives the clearest example of why AmigaOS can't have MP...