What is memory protection and why is it so hard to implement for the AmigaOS?

[psxphill]

Just what the world needs, an OS that behaves like facebook :lol:

Facebook isn't that bad. It's more like http://www.imdb.com/title/tt0196217/

[Fats]

@Fats

Such MP would be quite pointless to be honest (any non-MP app could easily nuke the whole OS or other apps, including the MP aware ones).

As said in another reply MP programs are protected from non-MP programs. When a non-MP program crashes the whole non-MP space may need to be cleaned-up. Call that a sandbox if you want, but it is different to most other proposals where the amiga(-like) sandbox runs on top of the core system which in my eyes is MP but not amiga-like.

How would the OS know if the caller if MP aware?

The executable is marked, executables without the mark won't be able to get MP. During task switch the OS knows if the task to switch to has MP or not.

What about 3rd party libraries the app might use? (remember that with AOS library stuff executes directly at the callers context).

MP programs that use a library function that violates the MP policies will get killed.

If one thing would use the flags wrong it'd lead to very hard to spot bugs.

No it would lead to MP programs getting killed like in any other MP OS.

You also forget that in order to have MMU access limitations to certain memory areas they need to be aligned by page size (start & len). With a shared global memory space OS this leads to massive address space fragmentation. And not all memory is allocated via memory pools.

Memory not allocated with the pools will be either fully public or private. I'll need to test what works best. The memory surely won't be protected and shareable with other programs at the same time.

 In short: You'd need a massive overhaul of the APIs and rewrite the apps anyway.

Who said it was easy ?
I think the big mistake made during the MP discussion was that people wanted to make a magical change to the OS and make all programs protected. I don't see this as possible. What I disagree with is that you need massive changes to the API to be able to introduce MP.
I also agree that programs that want to use MP will need to be changed and I need to test how involved these changes will be but ATM I think it is much less effort then lead to believe by some people.

The issue of MP has been debated for at least 15 years. The best you can get is the mmu.library protected executables. It protects code and read only data. Anything beyond that would require massive overhaul and API redesign, plus app rewriting.

I did not say it was easy, all I wanted to say that you can have almost identical API with extensions so program could protect their own memory and other memory could still be used to communicate with other programs in an amiga way.

From time to time some individuals come up with claims of generic trouble free MP for amigaos. Needless to say none of them have ever delivered. Some nameless OS even falsely claims to have MP while it really doesn't. Hint: The same OS was supposed to get automatic stack enlargement.

Was that system called pOS ?

Why this MEMF_PUBLIC insanity? There should be no reason why application should use this stupid brainfart. Remove memory usage restrictions and you are closer to real MP.

Of course all software I have written for Amiga allocates data on the stack and never cares about MEMF_PUBLIC. Believe it or not this coding practise works just fine. It works just fine even on Linux and Windows.

Then you are not implementing an amiga OS anymore IMHO. One thing - again IMHO - that at least is needed for an amiga-like OS is message passing by pointer passing and for that to succeed you need to have single address for the memory accessible in between different protection entities.

Also to be able to have stack extension no data that is to be shared may be allocated from the stack. This way the stack would be private memory space and it could have the same vritual address or overlapping addresses for different programs.

It could be done using UAE kind of solution. Run every program in its own sandbox.

No, because you can't send messages in between the sandboxes.

Yes, and that's just on the WWW! The usenet debates/flames about Amiga MP and resource tracking went back almost to the beginning. People have been thinking about it for almost a quarter century, and as Piru has pointed out, no one has delivered yet.

Unfortunately there are a lot of other AROS things to do and I have very limited time for programming. You'll need another quarter century probably until it is available or I have found out it is unmanageable.

Probably best to delay this discussion until the time is come I can actually start programming and prove the non-believers wrong .

greets,
Staf.

[Piru]

As said in another reply MP programs are protected from non-MP programs. When a non-MP program crashes the whole non-MP space may need to be cleaned-up. Call that a sandbox if you want, but it is different to most other proposals where the amiga(-like) sandbox runs on top of the core system which in my eyes is MP but not amiga-like.

I'll be very interested to see that in practice.

The executable is marked, executables without the mark won't be able to get MP. During task switch the OS knows if the task to switch to has MP or not.

Not all tasks/processes originate from binaries via execution. You have some thinking to do.

MP programs that use a library function that violates the MP policies will get killed.

Which means all but some very few existing libraries will be useless.

No it would lead to MP programs getting killed like in any other MP OS.

So you're also enforcing some new magical testing scheme that'll test all the code for all possible execution paths? Amazing!

Memory not allocated with the pools will be either fully public or private. I'll need to test what works best. The memory surely won't be protected and shareable with other programs at the same time.

You didn't address the problem I presented.

I think the big mistake made during the MP discussion was that people wanted to make a magical change to the OS and make all programs protected. I don't see this as possible. What I disagree with is that you need massive changes to the API to be able to introduce MP.

We disagree. I'm sure massive changes are necessary.

I also agree that programs that want to use MP will need to be changed and I need to test how involved these changes will be but ATM I think it is much less effort then lead to believe by some people.

Well you know what you have to do.

I did not say it was easy, all I wanted to say that you can have almost identical API with extensions so program could protect their own memory and other memory could still be used to communicate with other programs in an amiga way.

Theory and practice are a two separate things. I'll be expecting an update from you. Let me know when you have the thing working.

Also, would you comment this:

Indeed. The amiga APIs (and thus apps) are full of dependencies of single global memory space (you can add tons of various caller passed structures to system for instance, say PutMsg().. the struct Message will be added to foreign struct MsgPort. There are no strict rules how to allocate either. Many apps allocate them from stack, and that is perfectly legal with AmigaOS). It'd never work... Unless if you redesign the whole API, which in this plan was supposed to be avoided.

How are you going to handle messaging?

[AmigaEd]

I wouldn't let any Amiga powered on for weeks with lots of software running and important data unsaved. This isn't an academic discussion...

I understand what you're getting at but, I'm going to have to disagree with this to some extent. I did an experiment back in '93-'94 where I ran an A1000 continuously for nearly a year and half without a single problem. Never powered it down or gave it a three fingered vulcan nerve pinch. It was running several apps, including a BBS. So while there is risk, imho this needs to be considered in terms of opportunities vs. percentage of failure. I agree it's not academic, it has to be based on your real situation.

I wouldn't mind letting powered on MacOSX, Win2K+, Unix opened with lots of important stuff unsaved.

I can't speak in terms of MacOSX, but I can tell you that the company I work for has lost hundreds of thousands of dollars in trying to use NT, Win2K, and XP on systems requiring long term use and round the clock up time.

Cause unless there was a powercut I know my data would be back when I'll get back.

I'm not saying Amiga is better in this aspect but, I've been using Win based systems for years and none have ever given me a sense of data security. Quite the opposite in fact. I have done several studies on data loss with these WinOS based systems and the occurances of data loss is alarming especially when you factor out power loss events and operator caused conditions. If the Win based systems are so great in this respect then why is there such a large market for data recovery products and services. I've just spent the past year developing a methodology and software to work around many of the data loss short comings that exist with WinOS.

I don't want to have to choose the software I run because it may make me lose my work... nor do I want to wait before my HD led is off before powering my computer off, nor do I want to have to check I have enough graphics memory before running a (graphics) memory hungry software because the OS isn't capable of swapping: this is the task of the OS, not mine. The OS should (and will) take down any application trying to write outside its memory mapped,

Agreed! I can't wait till someone develops this!

the OS should close all apps and power the computer down when no app is writing to the disk

This doesn't sound like memory management to me.

[stefcep2]

I wouldn't let any Amiga powered on for weeks with lots of software running and important data unsaved. This isn't an academic discussion...

I wouldn't mind letting powered on MacOSX, Win2K+, Unix opened with lots of important stuff unsaved. Cause unless there was a powercut I know my data would be back when I'll get back.

I don't want to have to choose the software I run because it may make me lose my work... nor do I want to wait before my HD led is off before powering my computer off, nor do I want to have to check I have enough graphics memory before running a (graphics) memory hungry software because the OS isn't capable of swapping: this is the task of the OS, not mine. The OS should (and will) take down any application trying to write outside its memory mapped, the OS should close all apps and power the computer down when no app is writing to the disk, and the OS should automatically swap memory for my apps...

I think you're overstating the real world significance of these things.

This was acceptable 15 years ago. I think it's not today. Much like monotasking OS isn't acceptable, even for a phone... Would you use some DOS-like monotasking OS today ? don't you think having a GUI or not is an academic discussion ? after all you can do everything you want with just a simple console, right ?

Actually I've never used a single tasking OS.  Started on the Amiga and moved on to Win98SE-which crashed at least as much as my Amiga, suffered much more malware issues, but it did have MP, I suppose.

I don't think the CLI vs GUI argument is the same thing.

I understand the MP is A Good Thing.  Really I do, and I would rather have it than not.

But the REALITY is that millions of users for decades used AmigaOs 3.x without MP to develop god knows how many, apps, tools, utilitities  that didn't bring the OS down, or if they did it was rare, even when dozens ran at the same time.  Then look at the volume of  art, animation, music, video, DTP that was actually created which could not have happened if the apps kept crashing the OS.  And you can STILL do that with the same hardware and software today without MP.  So how much of a hindrance was the absence of MP in practice?  

Personally I think you're overstating the way users were on tenderhooks just waiting for the next imminent crash. It really didn't happen that way.  Most of the time.

[stefcep2]

Nonsense, many users left Amiga due to lack of MP when Internet became "the thing" to do with computers in early/mid 90ies - the lack of MP was also a major reason why Amiga users went on to create Linux and NetBSD for Amiga computers back then.

Many users left coz they couldn't play Doom.  At a user group we used to get 500-600 show up, and I honestly can't recall ANYONE saying, "I'm leaving Amiga beacsue it doesn't have MP."  

I can recall a pirate acquaintance (they always had the high spec hardware) with a warp engine A4000 downloading, uploading, IRC'ing, browsing burning on 3 burners in a scsi box, all the while connected through his 56 k modem, running a monster DOpus config, MCP and a host of hacks, cracks and kludges.  The thing was up for months at a time.   Don't think he missed the lack of MP, like most people.

And as for abusing Amigas online, I know that _I_ have "broken" into remote Amigas that due to malconfiguration and lack of MP meant a lot of fun for me, and maybe not so much fun for someone else.  The highlight was to dump prefs files into remote ENV: and watch the confused target rave about his problems on IRC :laughing:

No doubt it happened.  But 1.  How often and 2.  If I look at the patches released for Windows and internet software eg web browsers that runs on Windows that aim to patch another hole "where a remote user can access/add/change files on your computer", it would seem that the same risks are still there even with MP.  Oh and similar patches are being released for Linux and Linux internet software.

[runequester]

Im pretty sure people left because Commodore went bankrupt in early 94.

[stefcep2]

Im pretty sure people left because Commodore went bankrupt in early 94.

I think a lot did.  I remember all these souped Amiga's suddenly going for bargain prices in The Trading Post when Commodore went broke.  

What is surprising is that so many stuck around for so long afterwards.

[itix]

@Fats

Then you are not implementing an amiga OS anymore IMHO. One thing - again IMHO - that at least is needed for an amiga-like OS is message passing by pointer passing and for that to succeed you need to have single address for the memory accessible in between different protection entities.

Message passing is only minor detail in OS design. What about features like AddPort(), AddSemaphore(), AddTask() and other system calls? How do you protect system against rogue applications? I could add dummy uninitialized message port with invalid name tag. The entire OS could crash when it scans ports next time.

You can of course deprecate those functions. But then we are not talking about minor changes anymore. We are also having problem with layer hooks, global semaphores and other design flaws (from MP POV) added in Kickstart 2.0. Shared address space design is present everywhere.

Also to be able to have stack extension no data that is to be shared may be allocated from the stack. This way the stack would be private memory space and it could have the same vritual address or overlapping addresses for different programs.

And how are you going to implement stack extension with limited 32bit shared address space?

No, because you can't send messages in between the sandboxes.

You would obviously need an API extension to communicate safely with boxes. New applications would have to support it. Old applications could share sandbox.

[hardlink]

I did an experiment back in '93-'94 where I ran an A1000 continuously for nearly a year and half without a single problem. Never powered it down or gave it a three fingered vulcan nerve pinch. It was running several apps, including a BBS.

I believe it was Dr. Peter Kittel of CBM Germany/Escom that used to speak of how he ran his A1200 24/7/365 without problems or crashes. He was very careful with his system configuration, and carefully checked out any new software with enforcer/etc and did not run any software that was not clean. Sort of a software Whitelist, and it worked for him. But I'll bet he did not have as much fun as the rest of us

[vidarh]

Message passing is only minor detail in OS design. What about features like AddPort(), AddSemaphore(), AddTask() and other system calls? How do you protect system against rogue applications? I could add dummy uninitialized message port with invalid name tag. The entire OS could crash when it scans ports next time.

Start with total isolation per app. Break down just the isolation you need to get a functioning system, bridging messages where necessary.

Most Amiga apps don't need to share all that much information between them outside of a well defined set of messages to system services - ARexx messages, shared access to the same file systems and devices, windows being composited onto the same screens, input events. That's more or less it for the vast majority of applications.

Many of these can run as separate tasks/threads in the same "sandbox" as the applications that uses them. It is not necessary for the console.device to run in it's own address space on Amiga OS for example. Many other pieces of system code can reasonably run in the applications address space, further limiting the types of messages that needs to be possible to bridge to other processes.

So yes, you could add a dummy uninitialized message port with invalid name tag, and in the proposed model it would take down the tasks running in that one process for that one application, instead of taking down the entire OS. For "cross-process" message ports the kernel would necessarily need to do careful validation of all arguments.

[bloodline]

What you seem to be suggesting is "emumiga" for all apps, not just 68k ones...

[vidarh]

What you seem to be suggesting is "emumiga" for all apps, not just 68k ones...

Sort of, maybe... Emumiga (as well as Janus UAE and for that matter AROS hosted) is at least a good "proof of concept" that validate part of the idea.

Unlike Emumiga or Janus UAE, this sandboxing obviously wouldn't need to emulate the CPU or custom chips. It also would need far less bridging work since it doesn't have to deal with endianness issues. Compared to Emumiga the overhead would be trivial.

Both UAE and AROS hosted also already demonstrates the viability of part of the bridging (e.g. sound, filesystem access and input events are/can be bridged from the host, and can be shared across multiple instances of the hosted system). Janus UAE takes that further by also bridging individual windows to windows on the host instead of opening a single window for the emulation.

[zylesea]

If I understand things right, this is what MorphOS did with abox, and look what happened - abox is all there is, and little has changed.

Well not really. Abox is there but has more or less direct access to teh hw (drivers are inside teh ABOX). Th eprob is not Abox itself, but lack of Qbox and lack of drivers for virtualized hardware that Abox would need to use then if fully boxed.
But we all know that Qbox will probably never come. If the developers really had too much time I think it would be wise to consider a kind of "Qbox reloaded" to cope with an ISA switch, 64 bit migration, SMP, full resource tracking and maybe, maybee not MP and some other nice stuff. But probably the developers have a life and not the required time...

[itix]

Many of these can run as separate tasks/threads in the same "sandbox" as the applications that uses them. It is not necessary for the console.device to run in it's own address space on Amiga OS for example. Many other pieces of system code can reasonably run in the applications address space, further limiting the types of messages that needs to be possible to bridge to other processes.

But you would have to start new instance of console.device for each sandbox. And not just console.device but almost every other OS service. Drivers must get moved out of sandbox of course.

What about USB stack? I dont think it could stay inside this sandbox. But if it is outside sandbox how it runs? How do you modify USB stack to run there? Or is it just its own box? How about its own drivers?

It wont be easy

So yes, you could add a dummy uninitialized message port with invalid name tag, and in the proposed model it would take down the tasks running in that one process for that one application, instead of taking down the entire OS. For "cross-process" message ports the kernel would necessarily need to do careful validation of all arguments.

I dont think old PutMsg() would be suitable for cross process communication anymore so you would define new API to do that, anyway...

[vidarh]

But you would have to start new instance of console.device for each sandbox. And not just console.device but almost every other OS service. Drivers must get moved out of sandbox of course.

As with most things on AmigaOS, console.device is extremely lightweight, and contains minimal state. The vast majority of the "cost" of the console.device is the code pages - which are trivially shared - and the data structures for each individual console unit, which aren't shared no matter what. console.device is a perfect example of a device that don't need a separate sandbox, but can run in the same context as the application that opens it.

What about USB stack? I dont think it could stay inside this sandbox. But if it is outside sandbox how it runs? How do you modify USB stack to run there? Or is it just its own box? How about its own drivers?


It wont be easy

Why do you think it needs to be modified to run in its own sandbox? What is needed is a thin shim/bridge for messages between users of the USB stack in other address spaces and the stack itself. This is not much different than, say, the UAE or hosted AROS sharing of the host filesystem with the sandboxed apps.

And it's the same approach for any shared resource.

I dont think old PutMsg() would be suitable for cross process communication anymore so you would define new API to do that, anyway...

This can be dealt with for current apps either by adding information to the message ports, or by creating proxy ports, but it's possible it'd be more efficient to provide a new API for new apps.