What is memory protection and why is it so hard to implement for the AmigaOS?

[Piru]

@Fats

Such MP would be quite pointless to be honest (any non-MP app could easily nuke the whole OS or other apps, including the MP aware ones).

How would the OS know if the caller if MP aware? What about 3rd party libraries the app might use? (remember that with AOS library stuff executes directly at the callers context). If one thing would use the flags wrong it'd lead to very hard to spot bugs.

You also forget that in order to have MMU access limitations to certain memory areas they need to be aligned by page size (start & len). With a shared global memory space OS this leads to massive address space fragmentation. And not all memory is allocated via memory pools. In short: You'd need a massive overhaul of the APIs and rewrite the apps anyway.

The issue of MP has been debated for at least 15 years. The best you can get is the mmu.library protected executables. It protects code and read only data. Anything beyond that would require massive overhaul and API redesign, plus app rewriting.

From time to time some individuals come up with claims of generic trouble free MP for amigaos. Needless to say none of them have ever delivered. Some nameless OS even falsely claims to have MP while it really doesn't. Hint: The same OS was supposed to get automatic stack enlargement.

[Piru]

You can alleviate that by virtualising the address space, although that causes other problems.

Indeed. The amiga APIs (and thus apps) are full of dependencies of single global memory space (you can add tons of various caller passed structures to system for instance, say PutMsg().. the struct Message will be added to foreign struct MsgPort. There are no strict rules how to allocate either. Many apps allocate them from stack, and that is perfectly legal with AmigaOS). It'd never work... Unless if you redesign the whole API, which in this plan was supposed to be avoided.

[Piru]

The funny thing is that Microsoft team had similar problem in Windows 3.x era and they did it solving all those problems.

To be fair DOS virtualization is trivial compared to AmigaOS.

[Piru]

True. I don't compare DOS with AmigaOS but with Windows 3.x problem - mind you that they even had problem with different types of memory to solve which protected mode helped to fix. Anyway my point is that most of issues in term of theory has been solved already and this is strictly implementation problem in terms of producing code.

Not really, unless of course if you break the backwards compatibility and run old stuff in a sandbox that is.

There was a similar case in area of memory allocator when AmigaOS got it's own implementation of slab allocator and swap to make it more advance and efficient (which was a very wise move BTW).

Slab allocator is rather poor generic allocator, and I sure hope it isn't used as one. There seems to be great confusion about this, and everyone seems to repeat the slab mantra without really understanding the fundamentals. Slab cannot be used as a generic allocator, nor is it intended as such. I find it rather surprising that slab has been underlined so much in the new AmigaOS memory system, the actual kernel memory allocator is far more important. The original AmigaOS allocator used First Fit algoritm which was okay for the first couple of minutes of system use, but degraded to crawl after some time. It'd be rather important to know if this has been replaced with something better.

Two Level Segregate Fit on the other hand is way better, as demonstrated by TLSFMem patch and MorphOS TLSF memory system. TLSF is a generic memory allocator with constant allocation / deallocation time, regardless of the memory fragmentation.

[Piru]

As said in another reply MP programs are protected from non-MP programs. When a non-MP program crashes the whole non-MP space may need to be cleaned-up. Call that a sandbox if you want, but it is different to most other proposals where the amiga(-like) sandbox runs on top of the core system which in my eyes is MP but not amiga-like.

I'll be very interested to see that in practice.

The executable is marked, executables without the mark won't be able to get MP. During task switch the OS knows if the task to switch to has MP or not.

Not all tasks/processes originate from binaries via execution. You have some thinking to do.

MP programs that use a library function that violates the MP policies will get killed.

Which means all but some very few existing libraries will be useless.

No it would lead to MP programs getting killed like in any other MP OS.

So you're also enforcing some new magical testing scheme that'll test all the code for all possible execution paths? Amazing!

Memory not allocated with the pools will be either fully public or private. I'll need to test what works best. The memory surely won't be protected and shareable with other programs at the same time.

You didn't address the problem I presented.

I think the big mistake made during the MP discussion was that people wanted to make a magical change to the OS and make all programs protected. I don't see this as possible. What I disagree with is that you need massive changes to the API to be able to introduce MP.

We disagree. I'm sure massive changes are necessary.

I also agree that programs that want to use MP will need to be changed and I need to test how involved these changes will be but ATM I think it is much less effort then lead to believe by some people.

Well you know what you have to do.

I did not say it was easy, all I wanted to say that you can have almost identical API with extensions so program could protect their own memory and other memory could still be used to communicate with other programs in an amiga way.

Theory and practice are a two separate things. I'll be expecting an update from you. Let me know when you have the thing working.

Also, would you comment this:

Indeed. The amiga APIs (and thus apps) are full of dependencies of single global memory space (you can add tons of various caller passed structures to system for instance, say PutMsg().. the struct Message will be added to foreign struct MsgPort. There are no strict rules how to allocate either. Many apps allocate them from stack, and that is perfectly legal with AmigaOS). It'd never work... Unless if you redesign the whole API, which in this plan was supposed to be avoided.

How are you going to handle messaging?

[Piru]

There will be helper functions provided to ask for the memory region associated with a message port so you can allocate memory that has the right protection for the programs that will receive the messages through a port. Once this is OK you can just use the normal message handling functions. An alternative implementation is that during creation of a memory pool a list of ports can be provided and from then on memory allocated from this pool can be sent to these ports.

I count this as massive API change. Regardless how you do that you'll end up having to perform changes to the way application works, and you need to manually check all code for improper usage.

Then again if you're doing this on AROS you don't need to care about compatibility anyway.

[Piru]

In a MP program the stack can contain only data private to the program that can never be exported to other programs.

Making stack private will break a lot of things. Another large API change, requiring a lot of attention in all existing code.

[Piru]

It is a matter of terminology. Like BOOPSI, memory pools and AVL trees were not available in original OS 1.x and people needed to adapt their code to use them; memory pools with MP were not provided until now and people will need to adapt their code to use them.

None of these required mandatory changes. The old code functioned just fine. What you're proposing requires changes to the code, or the application will not work.

This is exactly what I'm talking about. Major API change so that you must go trough all your code and fix it manually.

[Piru]

imho the goal of aros, os4 and mos should be that.. guesstimate what the original team envisionned for that MP update and do it.

The whole contribution of the original team is some forward looking text file which throws in the air some things you should keep in mind in order to stay future system compatible. The instructions are incomplete and contradicting (for instance use of MEMF_PUBLIC). You can't blame them really, there's no way they could foresee the future development of multicore, memory management units in all CPUs etc.

Doing complete rework of the OS design to allow full memory protection and then running old apps inside UAE, what's bad in that you ask? If you're going to cook up your own design you'll in for 10+ year project reinventing the BSD/Linux wheel. Well, why bother with that, why not use use Linux and run UAE on top?

No, I personally don't feel like doing that.

Something like running your own lightweight kernel and userland and on top of linux kernel might make more sense. You wouldn't need to bother with the low level things then. Sounds much like Amithlon, with the exception that you'd replace the userland with newer one and perhaps not bother that much with HW emulation.

Everyone, feel free to steal that idea (albeit it isn't exactly original).

For now I'll rather keep working on MorphOS as it is now.

[Piru]

what i don't understand is why aros, os4 and mos users can't properly kill tasks and free their resources.

Global shared address space. Everything depends on it, from the very basics of the messaging system. From time to time some people claim to have invented a way to introduce a reliable and bulletproof memory protection scheme to amigoid system. It always ends the same way, all the sudden these people fall ominously quiet and the idea is forgotten for the next couple of years.

If you change the very basic concept of the global shared address space you will completely break all the existing APIs and applications. You'd need to have completely new set of applications, which would most likely be ports of open source apps. If so, how will it be any better than YALD?

[Piru]

well why not ?

Sure. Go ahead, I'll be waiting for your product release.

It's far easier to give advice than to actually produce something tangible.

And btw Apple used years on Mac OS X, and had budget of hundreds of millions.