How to move AROS forward

[Hans_]

@SamuraiCrow

SamuraiCrow wrote:
@HenryCase

Any time you see a structure being passed into a system function with an &structurename in C code, it is passing a pointer to an internal stucture in the application's memory to the operating system.  More importantly, if the operating system allocated the memory and passed back the pointer, the operating system would be the owner of that memory and would have to "change ownership" to the application and "change ownership" back to the operating system every time a pointer was passed back to it.  That would be a glacially slow operation.

Anything running in supervisor mode should have full access to all memory in the system. Thus, the OS kernel wouldn't have to change ownership back and forth. For non-kernel OS modules that run in user mode, they could run directly in the application's task/process; this means that there still wouldn't be any ownership change.

I think the answer to the issues that are often associated with memory protection are generally better addressed by using "safe" programming languages like Oberon and some versions of Basic that either don't directly support pointers in general, or have tight limitations on the use of arrays and other data structures to monitor them for overruns and NULL pointer values internally.

The problem with that is that it requires all software to play by the rules. Malicious code does not. Memory protection helps prevent malicious code from doing harm by restricting access to resources.

Hans

[Hans_]

SamuraiCrow wrote:

Hans_ wrote:
@SamuraiCrow

I think the answer to the issues that are often associated with memory protection are generally better addressed by using "safe" programming languages like Oberon and some versions of Basic that either don't directly support pointers in general, or have tight limitations on the use of arrays and other data structures to monitor them for overruns and NULL pointer values internally.

The problem with that is that it requires all software to play by the rules. Malicious code does not. Memory protection helps prevent malicious code from doing harm by restricting access to resources.

Hans

Memory protection doesn't stop malicious code either.  In a black-box operating system malicious code can overwrite anything that is in the same memory page as it is.  In a white-box operating system it can corrupt any system that the code has access to.

It's no panacea, but it does restrict the amount of damage that can be done. Private memory belonging to multiple processes shouldn't be on the same page. Using more memory than necessary for tiny programs is simply a price that you have to pay.

Let me put it this way, while programming languages/compilers could theoretically generate safe code, there's nothing to stop someone from deliberately generating nasty code. Being able to compromise the whole system with a few trivial instructions is not acceptable. Sure, you may say, just don't run code from people you don't trust. Ok, how do you know who to trust? Protecting the system is part of a modern OS' job.

Hans

[Hans_]

@HenryCase

The OS4 development team have looked at this, and decided that the only way to make it happen would be to break compatibility. Piru and Bloodline say so too. Trying to somehow discover who owns what memory, and who's still using it would be way to costly computationally, even if you could do it. The correct way forward is to create stricter rules about sharing memory, and that can't be done without breaking compatibility.

Whether it's AOS4.x, MOS, or AROS, the only way to get full memory protection would be to modify the API in a manner that breaks compatibility. Old apps would be run in a sand-boxed environment.

I'd be happy if they did break compatibility and provide a sand-box for old apps. While they're doing that, they might as well break other stuff to allow SMP, and to lock off user programs from accessing OS structures.

Hans

[Hans_]

HenryCase wrote:

"I'll go run the XOR opcode", so in other words I'm right.

The CPU has the ability to recognise an XOR opcode, and every other opcode it uses, and if part of an instruction isn't an opcode it is data. Therefore a CPU knows, after the 'decode' process, what is an opcode and what is an operand, correct? This requires just a yes or no answer.

The problem is that your data may just happen to match an op-code. Op-codes are restricted in number, but data can take any value. Thus, if you recognize an XOR op-code in memory, it could be an op-code, but it might not.

You could also check whether the memory is marked as executable, in which case you could probably confirm that something is indeed an op-code. Now what? That doesn't actually tell you what it's doing with data, who the data belongs to, etc.

Hans

[Hans_]

HenryCase wrote:

Hans wrote:
Now what? That doesn't actually tell you what it's doing with data, who the data belongs to, etc.

Don't opcodes always have the same operand requirements ('x' many operand bits for one opcode format, 'y' many operand bits for another format)? If so, couldn't you use this format to determine what the data used by an operand represented in non-binary form?

No. You could perform an XOR on any type of data, from text, through to graphics, and even pointers.

Hans

[Hans_]

HenryCase wrote:
Yes it can. Let's say you rewrite the compiler to add a "-MPAROS" compiler flag. Then whenever you try and compile programs with this flag it follows certain rules to make programs compatible with an MP kernel. The resultant code is different from compiling it normally, but crucially you wouldn't need to rewrite the application code yourself.

Let me give you an idea as to how difficult this would be. An MP version of the Amiga OS API would have to place restrictions on memory used in messages. All memory referenced in a message would have to be shared, or at least shared with the recipient. Old programs could use any memory, and might even allocate a message on the stack (which would be private). So the first step would be to re-allocate the message in shared memory.

So far so good. Unfortunately, this message could contain pointers to other memory. A compiler might know that some are pointers. However, coders have a tendency to do naughty things, and might write a pointer into a variable declared as an integer. Even if a coder does follow the rules, a pointer to an array or image, won't tell you how big that array or image is; and, if it's a dynamically allocated memory, you won't know at compile time (i.e., the compiler won't be able to do anything).

Now, to top everything off, this whole thing is recursive. The memory that is pointed to could point to other memory, which would point to other memory, etc.

This is why old apps would have to run in a sand-boxed environment. Old apps would collectively run in a shared memory space (so yes, they can interfere with one-another), and any interaction with the OS would have to go through a compatibility layer that would repackage calls to use the new API.

Hans
 

[Hans_]

@HenryCase

HenryCase wrote:
@bloodline and @Hans_
Thank you both again for informative posts. I think you misunderstand my plan, but that's understandable as I haven't explained it fully. Think of it like API translation at the compiler level. So instead of declaring AllocMem() as a pointer, the same attributes about the memory requirements (memory size, flags) get declared in a malloc kind of way. What I aim to do is take the API calls and turn them into ones that can support MP. Does this sound more reasonable?

The problem is that not all messages are allocated using AllocMem() (people should be using AllocVec()/AllocVecPooled() anyway). Messages that were allocated on the stack would not work with such a scheme. You are still going to encounter every problem that we've outlined. Sooner or later you are going to encounter things that cannot be determined at compile-time.

In theory it's all possible simply because we know that humans can do it. However, you will rapidly discover that human thought-processes are way more complex than anything that you can program. Moreover, it could take a human programmer months to make the transition, including extensive testing. Expecting a compiler to do this in seconds is unrealistic.

All of what I've written above ignores the fact that a developer would structure message contents differently in a full MP environment. Pointers to structures would be used less, strings would probably be dynamically allocated instead of just using a pointer to a constant, etc.

What exactly is wrong with allowing old software to be fully-allocated in shared-memory and run in a sand-box?

Hans

[Hans_]

bloodline wrote:

In an MP OS, one would define that the pointer returned by AllocMem() is for the use of the calling task only.
In AmigaOS the pointer returned by AllocMem() is free for any task to use, this is defined behaviour.

If you change the above definition you no longer have an AmigaOS compatible system, and everything would have to be rewritten, all the various subsystems would need to be altered in the way they work to the point that no programer would recognise the system as AmigaOS... It would probably look much more like Linux. And no Amiga Programs would compile for it anymore either...

It's not quite that bad. Amiga OS 4 programs can request private memory with the down-side that it's per task. There is no concept of multiple threads inside a process (or sub-processes, or whatever), so you can't perform direct sharing within a multi-threaded program. In a full MP OS, memory would be private by default, and you'd have to explicitly ask for shared memory. Henry is suggesting getting the compiler to insert the MEMF_SHARED flag (and/or whatever else woulld be used) whenever it's necessary. The problem is that finding out when it's necessary is incredibly complex.

There is no reason to completely drop message passing by reference. The only requirement would be that the message doesn't contain pointers to private memory.

The next problem we have is that function calls in AmigaOS are simple jumps from one executable to another... in an MP OS you have to trigger an interupt, so the machine can enter supervior mode, and then deal with the data as required.

Ok, now you've lost me. On all OSes, function calls to libraries are simple jumps. Sending messages between processes require an interrupt. However, that's exactly the same in Amiga OS. Amiga OS programs don't just suddenly start executing each other's code within their own task/process. If a message is sent, one task/process is put to sleep (at the interrupt), and the other is woken up by the OS so that it can respond.

Hans

[Hans_]

bloodline wrote:

Ok, now you've lost me. On all OSes, function calls to libraries are simple jumps.

I'm not talking about libraries, I'm talking about how you make a system cal to the Kernel. On AmigaOS Kernel calls are simple jumps the same as a library, but on a Real OS, that has a separate kernel space, you then need to use an interrupt to jump into Kernel space...

Ok.

Sending messages between processes require an interrupt. However, that's exactly the same in Amiga OS. Amiga OS programs don't just suddenly start executing each other's code within their own task/process. If a message is sent, one task/process is put to sleep (at the interrupt), and the other is woken up by the OS so that it can respond.

But calling the message function does not trigger a reschedule, AFAIR... thus does not interrupt...

The other task won't receive the message until a context switch (requiring an interrupt) occurs.

Anyway, my point was to show how separate address spaces actually work... Henry is missing a fundamental point of MP. It is this point we need to address.

There's no need for separate address-spaces. Full memory-protection can be achieved without it.

Hans

[Hans_]

HenryCase wrote:

itix wrote:
Again everything which is passed between applications and kernel must be public memory. Using private memory in Amiga is useless because such memory can not be passed to kernel calls because you dont know what kernel does with it. You can not trust that library calls can handle private memory.

So in an MP protected OS like, for example, FreeBSD, how do instructions/data get passed between applications and kernel?

What Itix wrote may be a bit confusing because the kernel can actually read/write to whatever memory it likes (it's running in supervisor mode, not user mode), so that is not an issue. The issue is how to pass data to non-kernel OS modules. For example, much of the x-windows system on Linux runs in user-mode, and hence cannot access memory that it doesn't own. Someone else can correct me if I'm wrong, but my understanding is that Linux/BSD copies a message's content from one process' memory space to another. As a consequence, there can be no pointers to other memory in that message.

Full memory-protection means that all the data in a message (including data referenced by pointers) has to be shared with (or copied to) the other process. Thus, if an Amiga OS program passes messages to others with pointers to it's internal data, that data would have to be in shared memory (or temporarily shared); therefore, this program wouldn't be protecting its internal data. Changing such an application to use full memory-protection would essentially be a redesign of its internals, not a quick change in memory flags.

Hans