What is memory protection and why is it so hard to implement for the AmigaOS?

[Fats]

It is not just that it breaks existing software. It is not possible implement memory protection without complete redesign of API.

I disagree here. I think it would be possible to have minor changes to the API and provide memory protection to programs that follow certain policies. By the latter I mean no data allocation on the stack that has to publicly available, obey the MEMF_PUBLIC/MEMF_PRIVATE meaning, etc.
Programs could in the executable or during start-up indicate that they follow these policies (with the consequence they may be killed when not following them). Programs not having this indicator won't be able to have MP.
For this to work you have to implement a MP that fits with the Amiga API. This means not trying to enforce the Windows/Linux MP with address spaces for each process/task but getting your inspiration from single address space operating systems and similar systems.
The main extension of the API would consist in extending the memory pools API so programs can indicate which other programs may access certain memory pools in read and/or write mode.

greets,
Staf.

[Fats]

@Fats

Such MP would be quite pointless to be honest (any non-MP app could easily nuke the whole OS or other apps, including the MP aware ones).

As said in another reply MP programs are protected from non-MP programs. When a non-MP program crashes the whole non-MP space may need to be cleaned-up. Call that a sandbox if you want, but it is different to most other proposals where the amiga(-like) sandbox runs on top of the core system which in my eyes is MP but not amiga-like.

How would the OS know if the caller if MP aware?

The executable is marked, executables without the mark won't be able to get MP. During task switch the OS knows if the task to switch to has MP or not.

What about 3rd party libraries the app might use? (remember that with AOS library stuff executes directly at the callers context).

MP programs that use a library function that violates the MP policies will get killed.

If one thing would use the flags wrong it'd lead to very hard to spot bugs.

No it would lead to MP programs getting killed like in any other MP OS.

You also forget that in order to have MMU access limitations to certain memory areas they need to be aligned by page size (start & len). With a shared global memory space OS this leads to massive address space fragmentation. And not all memory is allocated via memory pools.

Memory not allocated with the pools will be either fully public or private. I'll need to test what works best. The memory surely won't be protected and shareable with other programs at the same time.

 In short: You'd need a massive overhaul of the APIs and rewrite the apps anyway.

Who said it was easy ?
I think the big mistake made during the MP discussion was that people wanted to make a magical change to the OS and make all programs protected. I don't see this as possible. What I disagree with is that you need massive changes to the API to be able to introduce MP.
I also agree that programs that want to use MP will need to be changed and I need to test how involved these changes will be but ATM I think it is much less effort then lead to believe by some people.

The issue of MP has been debated for at least 15 years. The best you can get is the mmu.library protected executables. It protects code and read only data. Anything beyond that would require massive overhaul and API redesign, plus app rewriting.

I did not say it was easy, all I wanted to say that you can have almost identical API with extensions so program could protect their own memory and other memory could still be used to communicate with other programs in an amiga way.

From time to time some individuals come up with claims of generic trouble free MP for amigaos. Needless to say none of them have ever delivered. Some nameless OS even falsely claims to have MP while it really doesn't. Hint: The same OS was supposed to get automatic stack enlargement.

Was that system called pOS ?

Why this MEMF_PUBLIC insanity? There should be no reason why application should use this stupid brainfart. Remove memory usage restrictions and you are closer to real MP.

Of course all software I have written for Amiga allocates data on the stack and never cares about MEMF_PUBLIC. Believe it or not this coding practise works just fine. It works just fine even on Linux and Windows.

Then you are not implementing an amiga OS anymore IMHO. One thing - again IMHO - that at least is needed for an amiga-like OS is message passing by pointer passing and for that to succeed you need to have single address for the memory accessible in between different protection entities.

Also to be able to have stack extension no data that is to be shared may be allocated from the stack. This way the stack would be private memory space and it could have the same vritual address or overlapping addresses for different programs.

It could be done using UAE kind of solution. Run every program in its own sandbox.

No, because you can't send messages in between the sandboxes.

Yes, and that's just on the WWW! The usenet debates/flames about Amiga MP and resource tracking went back almost to the beginning. People have been thinking about it for almost a quarter century, and as Piru has pointed out, no one has delivered yet.

Unfortunately there are a lot of other AROS things to do and I have very limited time for programming. You'll need another quarter century probably until it is available or I have found out it is unmanageable.

Probably best to delay this discussion until the time is come I can actually start programming and prove the non-believers wrong .

greets,
Staf.

[Fats]

Which means all but some very few existing libraries will be useless.

On AROS you have the source code for most of the libs.

So you're also enforcing some new magical testing scheme that'll test all the code for all possible execution paths? Amazing!

No, the program is killed at the moment the non-compatible code wants to access a memory address that is protected. Not when it enters the code.

You didn't address the problem I presented.

You said that not all allocation is done through memory pools. I replied that protected memory regions have to be allocated through memory pools otherwise it won't have (full) memory protection. I also said that programs that want to use full memory protection will need to be adapted.
What am I missing ?

I'll be expecting an update from you. Let me know when you have the thing working.

If I ever come to it, I will do my work in AROS so you can find the announcements over there. I do have some other big projects that have higher priority: AROS ABI V1, sanitizing the AROS build system, unified build environment for MOS, AOS and AROS so in one go a binary is built for each of these OSes from the same source code, ...

How are you going to handle messaging?

There will be helper functions provided to ask for the memory region associated with a message port so you can allocate memory that has the right protection for the programs that will receive the messages through a port. Once this is OK you can just use the normal message handling functions. An alternative implementation is that during creation of a memory pool a list of ports can be provided and from then on memory allocated from this pool can be sent to these ports.

greets,
Staf.

[Fats]

@Fats
And how are you going to implement stack extension with limited 32bit shared address space?

In a MP program the stack can contain only data private to the program that can never be exported to other programs. This way all stacks of all programs could start on the same virtual address. (For security maybe some randomization of the start address could be introduced)
I am undecided if I should provide programmers the possiblity to choose for a fixed stack size with the advantage they can choose the protection of the memory from the stack. I think it is better to let all protected memory allocation happen through memory pools. In the latter case max. needed stack size should be much lower than on Linux/Windows.

I agree there may be some inefficiencies on 32bit machines and that you will not be able to effectively use the full 4GB adress space. How much you will loose will need to be tested and optimized.
For x86 I hope most of the people who need heavy duty programs would go 64 bit and for legacy m68k/PPC systems I think for most systems the ratio of virtual address space over physical present memory is big enough to allow some inefficiencies in the usage of the virtual address space.

greets,
Staf.

[Fats]

Then again if you're doing this on AROS you don't need to care about compatibility anyway.

With AROS I mean m68k AROS included with the ability to run old m68k code unmodified and not under emulation but without MP.

greets,
Staf.

[Fats]

I count this as massive API change. Regardless how you do that you'll end up having to perform changes to the way application works, and you need to manually check all code for improper usage.

It is a matter of terminology. Like BOOPSI, memory pools and AVL trees were not available in original OS 1.x and people needed to adapt their code to use them; memory pools with MP were not provided until now and people will need to adapt their code to use them.
If a MP programs want to send a message it needs to make sure the other side can read the memory it sends it. If two MP programs want to send messages to each other but protected from other programs than they need to have a way to get some memory region shared in between them. These facilities will always need to be provided if you stick to the message passing by pointer passing from amiga and will thus need API extension.
If you call that a major API change it is fine by me. If this means we agree, even for the better.

greets,
Staf.

[Fats]

So all arch of AROS need to suffer for a version that is not even being maintained?

For people who see it otherwise there are alternative projects like Anubis, DragonFlyBSD, .... If I wouldn't find source and backward compatibility important I would already have switched to DragonFlyBSD a long time ago.

And I don't think the other archs have to suffer because of it.

greets,
Staf.

[Fats]

What you're proposing requires changes to the code, or the application will not work.

No, what I am proposing is that the changes are required if the application is to have MP; the newly introduced functionality in the OS.
If they don't do the changes it will function just as before: without MP and being able to be killed by other bad behaving programs.
It's true that it is an all-or-nothing approach: if you decide to go MP you'll need to overhaul your whole program or it will crash at some point. And this is a difference with the examples I have given, I agree.
There exists some clever code analysis tools that with some adaptation should be able to help in finding most of the wrong struct allocation on stack and suspicious message passing with PutMsg()/SendMsg().

greets,
Staf.

[Fats]

This causes serious side effects. Consider following example:

void foobar(char *name)
{
   struct FileInfoBlock fib;
   TEXT buffer[108];
   BPTR fh;

   strcpy(buffer, name);
   strcat(buffer, ".txt");

   Printf("Open file %s\n", buffer);

   fh = Open(name, MODE_NEWFILE);
   Write(fh, buffer, sizeof(buffer));
   ExamineFH(fh, &fib);
}

Looks like you have to fix dos.library first.

I did not say that you could use libs without any change. And it will not only be dos but also exec, gfx, intuition, ...
In this example I don't see anything that can't be handled by dos internally, although the proof will be in the pudding.

 You are going to have other indirect implications. Think about this:

void PlaySound(BYTE *buffer, LONG bufferlength)
{
   io->Command = CMD_WRITE;
   io->Buffer = buffer;
   io->Length = bufferlength;
   DoIO(io);
}

If the buffer is allocated from stack this will cause problems. I agree.

Ooops? You must document what API calls can take data stored on the stack.

On Linux this is solved by having kernel code run under the kernel context without access to all memory without any restrictions. On a MP amiga OS this may sporadically be needed for efficiency reasons and limit the cases where API restrictions are needed on passed pointers to lib functions.
I would like to limit the code running under this 'god' context though and not evolve to a monolithic kernel; from the other side I am for some pragmatism and not some almost religious following of OS concepts like most of the microkernels guys have been doing.

This is why I dont like MEMF_PUBLIC crazyness.

To me it is not about enforcing MEMF_PUBLIC/MEMF_PRIVATE or not.
It is about how to put public data on stack and make the stack auto-extending. Do you have another solution for that ?

greets,
Staf.