What is memory protection and why is it so hard to implement for the AmigaOS?

[vidarh]

But you would have to start new instance of console.device for each sandbox. And not just console.device but almost every other OS service. Drivers must get moved out of sandbox of course.

As with most things on AmigaOS, console.device is extremely lightweight, and contains minimal state. The vast majority of the "cost" of the console.device is the code pages - which are trivially shared - and the data structures for each individual console unit, which aren't shared no matter what. console.device is a perfect example of a device that don't need a separate sandbox, but can run in the same context as the application that opens it.

What about USB stack? I dont think it could stay inside this sandbox. But if it is outside sandbox how it runs? How do you modify USB stack to run there? Or is it just its own box? How about its own drivers?


It wont be easy

Why do you think it needs to be modified to run in its own sandbox? What is needed is a thin shim/bridge for messages between users of the USB stack in other address spaces and the stack itself. This is not much different than, say, the UAE or hosted AROS sharing of the host filesystem with the sandboxed apps.

And it's the same approach for any shared resource.

I dont think old PutMsg() would be suitable for cross process communication anymore so you would define new API to do that, anyway...

This can be dealt with for current apps either by adding information to the message ports, or by creating proxy ports, but it's possible it'd be more efficient to provide a new API for new apps.

[itix]

Aaaeee, buggy forum software deleted my posting.

So only short question now: if USB stack mounts a USB masssstorage device woudnt filesystem for that USB drive run in UBS sandbox context?

[vidarh]

Aaaeee, buggy forum software deleted my posting.

So only short question now: if USB stack mounts a USB masssstorage device woudnt filesystem for that USB drive run in UBS sandbox context?

That's an implementation detail / performance vs. stability vs. development time tradeoff. But yes, that's one option. Another option is for the USB stack to be modified to spawn the filesystem in a separate sandbox and communicate via a cross-sandbox port.

Consider this: On a typical Linux system, almost all filesystems (except FUSE filesystems, and they're not used much) and all drivers run in the kernel context with 100% access to the entire system. Only applications are segregated. That's enough to get sufficient stability.

While the Amiga model in theory lends itself well to separating filesystems from the core OS, I think filesystems are less likely to affect system stability than random apps a user installs - for starters there are fewer of them, and they receive more widespread testing since if they fail they not only impact the system uptime but potentially corrupt data...

In other words, while the purist in me would love to see every task in it's own address space, segregated from each other, the pragmatist in me sees that that's neither necessary nor practical...

It's down to finding a happy medium that provides enough protection for stability on par with modern OS's and little enough to keep the 'cost' (in developer time and overhead) down.

[Fats]

Which means all but some very few existing libraries will be useless.

On AROS you have the source code for most of the libs.

So you're also enforcing some new magical testing scheme that'll test all the code for all possible execution paths? Amazing!

No, the program is killed at the moment the non-compatible code wants to access a memory address that is protected. Not when it enters the code.

You didn't address the problem I presented.

You said that not all allocation is done through memory pools. I replied that protected memory regions have to be allocated through memory pools otherwise it won't have (full) memory protection. I also said that programs that want to use full memory protection will need to be adapted.
What am I missing ?

I'll be expecting an update from you. Let me know when you have the thing working.

If I ever come to it, I will do my work in AROS so you can find the announcements over there. I do have some other big projects that have higher priority: AROS ABI V1, sanitizing the AROS build system, unified build environment for MOS, AOS and AROS so in one go a binary is built for each of these OSes from the same source code, ...

How are you going to handle messaging?

There will be helper functions provided to ask for the memory region associated with a message port so you can allocate memory that has the right protection for the programs that will receive the messages through a port. Once this is OK you can just use the normal message handling functions. An alternative implementation is that during creation of a memory pool a list of ports can be provided and from then on memory allocated from this pool can be sent to these ports.

greets,
Staf.

[Piru]

There will be helper functions provided to ask for the memory region associated with a message port so you can allocate memory that has the right protection for the programs that will receive the messages through a port. Once this is OK you can just use the normal message handling functions. An alternative implementation is that during creation of a memory pool a list of ports can be provided and from then on memory allocated from this pool can be sent to these ports.

I count this as massive API change. Regardless how you do that you'll end up having to perform changes to the way application works, and you need to manually check all code for improper usage.

Then again if you're doing this on AROS you don't need to care about compatibility anyway.

[Fats]

@Fats
And how are you going to implement stack extension with limited 32bit shared address space?

In a MP program the stack can contain only data private to the program that can never be exported to other programs. This way all stacks of all programs could start on the same virtual address. (For security maybe some randomization of the start address could be introduced)
I am undecided if I should provide programmers the possiblity to choose for a fixed stack size with the advantage they can choose the protection of the memory from the stack. I think it is better to let all protected memory allocation happen through memory pools. In the latter case max. needed stack size should be much lower than on Linux/Windows.

I agree there may be some inefficiencies on 32bit machines and that you will not be able to effectively use the full 4GB adress space. How much you will loose will need to be tested and optimized.
For x86 I hope most of the people who need heavy duty programs would go 64 bit and for legacy m68k/PPC systems I think for most systems the ratio of virtual address space over physical present memory is big enough to allow some inefficiencies in the usage of the virtual address space.

greets,
Staf.

[Fats]

Then again if you're doing this on AROS you don't need to care about compatibility anyway.

With AROS I mean m68k AROS included with the ability to run old m68k code unmodified and not under emulation but without MP.

greets,
Staf.

[Piru]

In a MP program the stack can contain only data private to the program that can never be exported to other programs.

Making stack private will break a lot of things. Another large API change, requiring a lot of attention in all existing code.

[Fats]

I count this as massive API change. Regardless how you do that you'll end up having to perform changes to the way application works, and you need to manually check all code for improper usage.

It is a matter of terminology. Like BOOPSI, memory pools and AVL trees were not available in original OS 1.x and people needed to adapt their code to use them; memory pools with MP were not provided until now and people will need to adapt their code to use them.
If a MP programs want to send a message it needs to make sure the other side can read the memory it sends it. If two MP programs want to send messages to each other but protected from other programs than they need to have a way to get some memory region shared in between them. These facilities will always need to be provided if you stick to the message passing by pointer passing from amiga and will thus need API extension.
If you call that a major API change it is fine by me. If this means we agree, even for the better.

greets,
Staf.

[dammy]

Then again if you're doing this on AROS you don't need to care about compatibility anyway.

Amen!

[dammy]

With AROS I mean m68k AROS included with the ability to run old m68k code unmodified and not under emulation but without MP.

So all arch of AROS need to suffer for a version that is not even being maintained?

[Fats]

So all arch of AROS need to suffer for a version that is not even being maintained?

For people who see it otherwise there are alternative projects like Anubis, DragonFlyBSD, .... If I wouldn't find source and backward compatibility important I would already have switched to DragonFlyBSD a long time ago.

And I don't think the other archs have to suffer because of it.

greets,
Staf.

[Piru]

It is a matter of terminology. Like BOOPSI, memory pools and AVL trees were not available in original OS 1.x and people needed to adapt their code to use them; memory pools with MP were not provided until now and people will need to adapt their code to use them.

None of these required mandatory changes. The old code functioned just fine. What you're proposing requires changes to the code, or the application will not work.

This is exactly what I'm talking about. Major API change so that you must go trough all your code and fix it manually.

[Fats]

What you're proposing requires changes to the code, or the application will not work.

No, what I am proposing is that the changes are required if the application is to have MP; the newly introduced functionality in the OS.
If they don't do the changes it will function just as before: without MP and being able to be killed by other bad behaving programs.
It's true that it is an all-or-nothing approach: if you decide to go MP you'll need to overhaul your whole program or it will crash at some point. And this is a difference with the examples I have given, I agree.
There exists some clever code analysis tools that with some adaptation should be able to help in finding most of the wrong struct allocation on stack and suspicious message passing with PutMsg()/SendMsg().

greets,
Staf.

[itix]

In a MP program the stack can contain only data private to the program that can never be exported to other programs. This way all stacks of all programs could start on the same virtual address.

This causes serious side effects. Consider following example:

void foobar(char *name)
{
   struct FileInfoBlock fib;
   TEXT buffer[108];
   BPTR fh;

   strcpy(buffer, name);
   strcat(buffer, ".txt");

   Printf("Open file %s\n", buffer);

   fh = Open(name, MODE_NEWFILE);
   Write(fh, buffer, sizeof(buffer));
   ExamineFH(fh, &fib);
}

Looks like you have to fix dos.library first. You are going to have other indirect implications. Think about this:

void PlaySound(BYTE *buffer, LONG bufferlength)
{
   io->Command = CMD_WRITE;
   io->Buffer = buffer;
   io->Length = bufferlength;
   DoIO(io);
}

Ooops? You must document what API calls can take data stored on the stack. This is why I dont like MEMF_PUBLIC crazyness.

[Fats]

This causes serious side effects. Consider following example:

void foobar(char *name)
{
   struct FileInfoBlock fib;
   TEXT buffer[108];
   BPTR fh;

   strcpy(buffer, name);
   strcat(buffer, ".txt");

   Printf("Open file %s\n", buffer);

   fh = Open(name, MODE_NEWFILE);
   Write(fh, buffer, sizeof(buffer));
   ExamineFH(fh, &fib);
}

Looks like you have to fix dos.library first.

I did not say that you could use libs without any change. And it will not only be dos but also exec, gfx, intuition, ...
In this example I don't see anything that can't be handled by dos internally, although the proof will be in the pudding.

 You are going to have other indirect implications. Think about this:

void PlaySound(BYTE *buffer, LONG bufferlength)
{
   io->Command = CMD_WRITE;
   io->Buffer = buffer;
   io->Length = bufferlength;
   DoIO(io);
}

If the buffer is allocated from stack this will cause problems. I agree.

Ooops? You must document what API calls can take data stored on the stack.

On Linux this is solved by having kernel code run under the kernel context without access to all memory without any restrictions. On a MP amiga OS this may sporadically be needed for efficiency reasons and limit the cases where API restrictions are needed on passed pointers to lib functions.
I would like to limit the code running under this 'god' context though and not evolve to a monolithic kernel; from the other side I am for some pragmatism and not some almost religious following of OS concepts like most of the microkernels guys have been doing.

This is why I dont like MEMF_PUBLIC crazyness.

To me it is not about enforcing MEMF_PUBLIC/MEMF_PRIVATE or not.
It is about how to put public data on stack and make the stack auto-extending. Do you have another solution for that ?

greets,
Staf.