Coldfire AGAIN

[HenryCase]

bloodline wrote:
MMU's are hard to get your head around at first but some solid study and a lot of running it out on paper with a pencil will give you a good idea of the issues here...

I am studying memory protection, but as you failed to grasp my ideas correctly allow me to try and explain them again...

You can't just switch off the MMU once it is on... the whole integrity of the memory structure is maintained by this device... Switching it off would leave the memory in effectly a random jumble of 4k blocks that used to mean something...

Yes you can. Imagine this scenario. You have two kernels available MP-Exec (Memory Protected Exec) and Exec (the standard no memory protection version). You've started running a program, but need it to run faster. When you give the command to switch from MP-Exec to Exec (I know this would remove memory protection for all processes), what effectively happens is that the MMU updates the 'virtual' memory addresses with the real memory addresses (like saying to the program 'here's where your data was'). Once this process is complete a kernel switcher program takes over, freezing all processes and keeping the memory as it was (kind of like when you put a laptop into hibernate). The kernels are switched and the processes and memory are unfrozen. The programs don't see any difference because the APIs of MP-Exec and Exec give the same results.

I really can't see a problem with what I've described, perhaps you can?

One of the great things about AmigaOS is that is places no restrictons on what is passed in an exec message... In effect the OS has no idea what the programs are sending to each other. (it's the same problem as the parasite emulator we discussed before).

But this can be changed by adding an MMU layer to the kernel and updating the kernel functions, so it's not an issue. Of course no memory protection gives you more freedom, what I am describing is a way to add memory protection for those who want it (which seems to be a lot of people).

bloodline wrote:
Now here is where the problems begin... You have two tasks, both have their own address spaces... one task sends a pointer to a data block to another task in a message... that pointer is meaningless to the receiving task, since the pointer is either not vaild in its address space or pointer for somthing completely different!

As the MMU is dishing out memory, it knows what kind of data will be valid for the space. So, for example, if you try to pass a message suitable for a  graphics routine into a memory space designated for sound functions then the MMU will not allow it. This is just one example, there will be other protection measures in place, such as if a program crashes you could limit the crash to its own allocated memory space, seeing as all reading and writing of memory goes through the MMU.

bloodline wrote:
The only way around this is to have a public pool of memory mapped to the same address window in all task address spaces... but since the public/private flags were ignored this can't work.

FYI, that's not what I'm trying to propose.

bloodline wrote:
I don't understand... The way memory protection would work is that each task gets it's own address space... in effect each task gets up to 4gig of address space to itself. I shall expalin the reprocussions of this later...

Don't understand? Okay, this is how it works...

When an API call requests a memory space (doesn't matter if it has flagged anything) the MMU layer of the OS looks for a suitable space to store that memory and then tells the API call (I stored your data 0x001 here: 1F2C, or whatever). The point is, that doesn't have to be where the data is stored, the app just has to think that's where it is stored. The MMU has a look up table linking the 'virtual' location with the real memory location. When the memory stored at 1F2C gets called the MMU then fetches the real data...

My laptop battery is about to die. I do have more to say but I'm hoping the penny has dropped by now. I'll be back later, feel free to comment on what I've said so far.

[Karlos]

...what effectively happens is that the MMU updates the 'virtual' memory addresses with the real memory addresses (like saying to the program 'here's where your data was').

...

I really can't see a problem with what I've described, perhaps you can?

How do you take, say, 16 4KiB pages scattered across the 4G physical address space that an application requested and originally thought was one contiguous 64KiB lump of memory and tell it "here is where your data was" ?

A single allocation of memory on a VM system using an MMU that an application uses a single pointer to refer to can translate into many unrelated chunks of genuine physical memory. You can't assume contiguous address mapped memory is contiguous in physical RAM.

[Kronos]

@HenryCase

Problem is the OS can't decide wether one task made a legitemate request for data owned by another (lets say stuff running inside the input.device-task) or if it is just some roque pointer .....

Next idea is usually to just copy system-structures into each tasks private part of memory, but once you take a look into the interlinked (and open) system-structures you'll notice that ain't pratical either.

[Einstein]

AeroMan wrote:
Ok, I recognize it was my fault to express my frustration with Windows problems, and I apologize for that, even if I don't agree with you.

Reality and facts ignore personal opinions, sorry to say.

Can we go back to Coldfire ?  :-D It was getting really interesting...

You are still free to do so.

HenryCase wrote:

Not true. AmigaOS4, MorphOS, AROS64 all have at least partial memory protection.

Yea, remember the (tiny) shield and the samurai ?

I am currently investigating how easy it will be to add full memory protection to AROS (without throwing away AmigaOS 3.1 API compatibility),

No can do without an emulation/compatibilty layer, you still need an emulation layer #?UAE style if you want a portable yet amiga compatible OS, I belive bloodline gave you the reason why you can't just add MP to exec and think everythings gonna be alright.

and so I asked you which version you had used so I could get an idea of the stability compared to the other memory protection options.

Like I pointed out, it's *irrelevant*, if a buggy app copmtabile with OS 3.1, 2.1, and 1.3 runs in these it will bring'em down regardless how "stable" (bugless) any particular OS version is.

So I'll ask again, which version of AmigaOS were you using?

Last time it was 1.3, 2.0 and 3.something AIAB, when several soundtrackers sent the OS to hell. Offcourse those Soundtrackers werent compatible with the higher versions, but my point remains. Since then I havent touched any of'em.

Karlos wrote:
Memory protection exists to clean up the mess of bad coders...

*runs away*

And traffic lights exist to prevent the mess that would arise due to bad and "bad" drivers.

[HenryCase]

Karlos wrote:
How do you take, say, 16 4KiB pages scattered across the 4G physical address space that an application requested and originally thought was one contiguous 64KiB lump of memory and tell it "here is where your data was" ?

By arranging it into a 64KiB lump before you give memory control back to the program.

Karlos wrote:
A single allocation of memory on a VM system using an MMU that an application uses a single pointer to refer to can translate into many unrelated chunks of genuine physical memory. You can't assume contiguous address mapped memory is contiguous in physical RAM.

Give me an example of when you'd use a pointer to address more than one memory location so I can explain how its done.

[Hans_]

Kronos wrote:
"the existing messaging system" would have to be updated to 64Bit in order to make full use of it, but that would require a recompile.


Wether it would actually do anything in regard to MP is questionable and would atleast require further measures (which will break source-compability too).

Updating to 64-bit would only need to be done on a 64-bit processor. Seeing as that jump is going to break stuff anyway, migration to a 64-bit processor would be a good time at which to make the necessary changes.

At a bare minimum, memory for messages should be marked as shared. Even better, message allocation/deallocation functions should be created that do this. What I was trying to say is that we wouldn't need a completely new message passing system. There would be additional restrictions, but the fundamentals would remain the same. The differences would be enough that old apps would have to run in a sandbox.

Hans

[HenryCase]

Kronos wrote:
@HenryCase

Problem is the OS can't decide wether one task made a legitemate request for data owned by another (lets say stuff running inside the input.device-task) or if it is just some roque pointer .....

I am not asking it to. I am asking the OS to make some limits on the commands that are physically not possible, like trying to put a 6-bit instruction into a 3-bit space. It was just one example of how the memory layer would keep the system stable.

Kronos wrote:
Next idea is usually to just copy system-structures into each tasks private part of memory, but once you take a look into the interlinked (and open) system-structures you'll notice that ain't pratical either.

Please expand on that point so I can understand it.

[Louis Dias]

I see that in the old OS, the memory protection flags were ignored...but isn't the real question - were they actually set?

If they were actually set, couldn't the OS allocate a block of public memory available to all applications as well as private memory for each individual app...

If the some apps didn't set the flag in the call, wouldn't it be simple to find those calls and patch them by flipping a bit.  Can't be hard to see where the app crashes...and find the offending call...

This could be done with a "debug" Amiga emulator on x86 where you can poke around and step through stuff...

I know from the manuals I read, I read to always set the flag appropriately, but I only scratched the surface of Amiga programming 11 years ago...so I probably have things quite off...

[AJCopland]

HenryCase wrote:

Karlos wrote:
How do you take, say, 16 4KiB pages scattered across the 4G physical address space that an application requested and originally thought was one contiguous 64KiB lump of memory and tell it "here is where your data was" ?

By arranging it into a 64KiB lump before you give memory control back to the program.

Wow you can do that? (note: this is sarcasm)
Ok technically it might be possible to go an find all of those fragments of memory allocated to a process using the MMU somehow but even once you've found them and "give them back" the addresses that you "give back" to the program when you congeal that memory aren't going to be the same ones as it originally had when they were allocated under the MMU scheme.

Karlos wrote:
A single allocation of memory on a VM system using an MMU that an application uses a single pointer to refer to can translate into many unrelated chunks of genuine physical memory. You can't assume contiguous address mapped memory is contiguous in physical RAM.

Give me an example of when you'd use a pointer to address more than one memory location so I can explain how its done.

Striding through an array :-D

With all due respect Henry both bloodline and karlos are trying to explain how MMUs work from a position of knowing, at least from a code point-of-view, how they function. So some of your replies have read as being rather rude.

Andy

[HenryCase]

HenryCase wrote:
My laptop battery is about to die. I do have more to say but I'm hoping the penny has dropped by now. I'll be back later, feel free to comment on what I've said so far.

Continuing from where I left off...

Thought of an analogy about API calls that may be useful. In a restaurant, think of the API calls as the customers and the memory management and kernel as the waiter and chef. The customer can make any number of rediculous requests about what they'd like to eat, but in a restaurant it's the chef and waiter that decides what they're having. A non memory protected OS is like a restuarant without a waiter, the customers have to go into the kitchen to get their food, and can make a mess. If you put the waiter as a barrier between the kitchen and the customers then they can't make a mess in the kitchen. Hmm, must have food on the brain!

The point is API calls are not in control of the system, they are only means to make requests to that system. As long as the waiter delivers the same food that the chef was providing before the customer is not going to complain. Capiche?

Let me set a challenge. I've hopefully explained a little about how this memory abstraction layer will work. I'm going to stop calling it a MMU now and call it a MAU (Memory Allocation Unit) because although I am confident my ideas will work I don't know if this is how MMU's traditionally work.

My challenge is this. I have a program running which has got an important string "ABCDEF" stored in memory at real location "0x001" and virtual location "0x0FF". Using your knowledge of the Amiga's API architecture, I want you to try and corrupt this data. I am confident that whatever you suggest can be dealt with without breaking AmigaOS compatibility, so try and prove me wrong.

[bloodline]

Hans_ wrote:

Kronos wrote:
"the existing messaging system" would have to be updated to 64Bit in order to make full use of it, but that would require a recompile.


Wether it would actually do anything in regard to MP is questionable and would atleast require further measures (which will break source-compability too).

Updating to 64-bit would only need to be done on a 64-bit processor. Seeing as that jump is going to break stuff anyway, migration to a 64-bit processor would be a good time at which to make the necessary changes.

At a bare minimum, memory for messages should be marked as shared. Even better, message allocation/deallocation functions should be created that do this. What I was trying to say is that we wouldn't need a completely new message passing system. There would be additional restrictions, but the fundamentals would remain the same. The differences would be enough that old apps would have to run in a sandbox.

Hans

Hans!! There is nothing wrong with the message system, the problem is what is sent in those messages! The OS doesn't know what they contain and they could contain anything! :-)

[HenryCase]

AJCopland wrote:
Wow you can do that? (note: this is sarcasm)

Think of it like HD defragmentation if it helps you understand it.

AJCopland wrote:
With all due respect Henry both bloodline and karlos are trying to explain how MMUs work from a position of knowing, at least from a code point-of-view, how they function. So some of your replies have read as being rather rude.

I'm not trying to be rude, but it's hard to maintain composure when you are being misunderstood. As I've just mentioned, lets call my idea an MAU (Memory Allocation Unit) as apposed to a MMU as I don't know if what I'm describing is your daddy's memory management system, if you catch my drift.

[AJCopland]

HenryCase wrote:
My challenge is this. I have a program running which has got an important string "ABCDEF" stored in memory at real location "0x001" and virtual location "0x0FF". Using your knowledge of the Amiga's API architecture, I want you to try and corrupt this data. I am confident that whatever you suggest can be dealt with without breaking AmigaOS compatibility, so try and prove me wrong.

10: program#1 has address 0x0FF
20: tells program#2 that the string it wants is at 0x0FF
30: program#2 accesses 0x0FF and catches a waiter whacking off into program#1's soup...

...they never eat at that restaurant again :crazy:

[Einstein]

HenryCase wrote:

As I've just mentioned, lets call my idea an MAU (Memory Allocation Unit) as apposed to a MMU as I don't know if what I'm describing is your daddy's memory management system, if you catch my drift.

You could retain MMU, and call it Memory Mapping Unit

[bloodline]

HenryCase wrote:

Karlos wrote:
How do you take, say, 16 4KiB pages scattered across the 4G physical address space that an application requested and originally thought was one contiguous 64KiB lump of memory and tell it "here is where your data was" ?

By arranging it into a 64KiB lump before you give memory control back to the program.

Memory allocated in a Memory Protected environment is built up from small blocks of RAM... they are usually 4k in size (this is convenient on a 32bit system) and they are called pages... The MMU can take  pages from anywhere in the Physical address space, and it puts them together into a virtual address space that the task sees as a single contiguous area of ram. This is why MP systems don't suffer from RAM fragmentation like the Amiga does. What you are suggesting is a serious amount of house keeping... MP is expensive enough in terms of RAM and CPU Cycles... in order to rebuild the virtual address space would require probably another 4gig on top of your exisitng ram!!!

You really do need to read about MMU's I suggest the x86 MMU as it is the most logical IMHO... I would be more than happy to help answer any questions!

Karlos wrote:
A single allocation of memory on a VM system using an MMU that an application uses a single pointer to refer to can translate into many unrelated chunks of genuine physical memory. You can't assume contiguous address mapped memory is contiguous in physical RAM.

Give me an example of when you'd use a pointer to address more than one memory location so I can explain how its done.

In MP system a contiguous block of Virtual memory is more than one Physical memory location. :-)

[bloodline]

HenryCase wrote:

HenryCase wrote:
My laptop battery is about to die. I do have more to say but I'm hoping the penny has dropped by now. I'll be back later, feel free to comment on what I've said so far.

Continuing from where I left off...

Thought of an analogy about API calls that may be useful. In a restaurant, think of the API calls as the customers and the memory management and kernel as the waiter and chef. The customer can make any number of rediculous requests about what they'd like to eat, but in a restaurant it's the chef and waiter that decides what they're having. A non memory protected OS is like a restuarant without a waiter, the customers have to go into the kitchen to get their food, and can make a mess. If you put the waiter as a barrier between the kitchen and the customers then they can't make a mess in the kitchen. Hmm, must have food on the brain!

The point is API calls are not in control of the system, they are only means to make requests to that system. As long as the waiter delivers the same food that the chef was providing before the customer is not going to complain. Capiche?

Let me set a challenge. I've hopefully explained a little about how this memory abstraction layer will work. I'm going to stop calling it a MMU now and call it a MAU (Memory Allocation Unit) because although I am confident my ideas will work I don't know if this is how MMU's traditionally work.

My challenge is this. I have a program running which has got an important string "ABCDEF" stored in memory at real location "0x001" and virtual location "0x0FF". Using your knowledge of the Amiga's API architecture, I want you to try and corrupt this data. I am confident that whatever you suggest can be dealt with without breaking AmigaOS compatibility, so try and prove me wrong.

Task A Stores "ABCDEF" at 0x0FF.
Task A sends a message to Task B.
The message simply contains 0x0FF.
Task B looks at Address 0x0FF and it doesn't find "ABCDEF".

Reason:
0x0FF are two different Physical memory locations to the separate tasks. infact Task B might even have some code at its 0x0FF, or anthing!!!

The OS has no idea that Physical address 0x0001 has to be mapped to the same place in both tasks... in fact it might not even be possible to map that physical address to the same virtual address!

Neither of the Tasks can ever know the Physical address, as their 0x0001 is mapped somewhere else... different for each task!!!