Coldfire AGAIN

[bloodline]

HenryCase wrote:

HenryCase wrote:
My laptop battery is about to die. I do have more to say but I'm hoping the penny has dropped by now. I'll be back later, feel free to comment on what I've said so far.

Continuing from where I left off...

Thought of an analogy about API calls that may be useful. In a restaurant, think of the API calls as the customers and the memory management and kernel as the waiter and chef. The customer can make any number of rediculous requests about what they'd like to eat, but in a restaurant it's the chef and waiter that decides what they're having. A non memory protected OS is like a restuarant without a waiter, the customers have to go into the kitchen to get their food, and can make a mess. If you put the waiter as a barrier between the kitchen and the customers then they can't make a mess in the kitchen. Hmm, must have food on the brain!

The point is API calls are not in control of the system, they are only means to make requests to that system. As long as the waiter delivers the same food that the chef was providing before the customer is not going to complain. Capiche?

Let me set a challenge. I've hopefully explained a little about how this memory abstraction layer will work. I'm going to stop calling it a MMU now and call it a MAU (Memory Allocation Unit) because although I am confident my ideas will work I don't know if this is how MMU's traditionally work.

My challenge is this. I have a program running which has got an important string "ABCDEF" stored in memory at real location "0x001" and virtual location "0x0FF". Using your knowledge of the Amiga's API architecture, I want you to try and corrupt this data. I am confident that whatever you suggest can be dealt with without breaking AmigaOS compatibility, so try and prove me wrong.

Task A Stores "ABCDEF" at 0x0FF.
Task A sends a message to Task B.
The message simply contains 0x0FF.
Task B looks at Address 0x0FF and it doesn't find "ABCDEF".

Reason:
0x0FF are two different Physical memory locations to the separate tasks. infact Task B might even have some code at its 0x0FF, or anthing!!!

The OS has no idea that Physical address 0x0001 has to be mapped to the same place in both tasks... in fact it might not even be possible to map that physical address to the same virtual address!

Neither of the Tasks can ever know the Physical address, as their 0x0001 is mapped somewhere else... different for each task!!!

[Hans_]

bloodline wrote:
Hans!! There is nothing wrong with the message system, the problem is what is sent in those messages! The OS doesn't know what they contain and they could contain anything! :-)

I agree. The thing that would change in a system with full MP is that programmers will have to allocate memory appropriately and be more careful about what's in the messages.

Hans

[AJCopland]

HenryCase wrote:

AJCopland wrote:
Wow you can do that? (note: this is sarcasm)

Think of it like HD defragmentation if it helps you understand it.

Oh I understand what you're saying perfectly clearly. An old version of an arcade game conversion that I worked on used a shared pool of memory instead of doing full blown networking (don't 'cos I can't tell you which). Each machine was given a physical ID (using jumpers) and this told them what address they were allowed to write into, and conversely what addresses other machines on the "network" would be writing into.

This situation is similar to the AmigaOS how? You ask. Well with AOS you've got a linear address space. Which means that to program#1 a pointer with the address 0x0000FEED is the same as program#2s address of 0x0000FEED.

Now at this point my knowledge of workbench/AOS gets wonky 'cos i've never done any coding in it.

The above means that programs could, and probably did, pass around the pointer addresses to data rather than actual data itself because since nothing ever set or respected the memory protection flags it was more efficient to pass a 4byte memory address than a larger data structure like a string.

Implementing memory protection or memory management et al at this stage would break all of those existing programs, and probably workbench as well because they'd no longer be able to communicate with either the underlying OS or the other programs like they used too as each program would be within its own 4GB protected memory space and so the pointer addresses that it was merrily passing around would no longer match those of the program on the recieving end.

EDIT: The relevence to the first paragraph i wrote is that if those arcade machines OS had memory protection then they'd never get each others messages because whilst they'd all be accessing their _own_ versions of the correct memory addresses the physical section of memory that they'd be writing too would be different from the physical address that the other machines were reading from :EDIT

Now that might all be bollox because i've had a long day.

I also just realised that this was the ColdFIRE thread and I only came in here in the hope that someone would be commenting on THAT topic. So I apologise profusely to everyone for continuing this off-topic subject.

Andy

[AJCopland]

bloodline wrote:
Task A Stores "ABCDEF" at 0x0FF.
Task A sends a message to Task B.
The message simply contains 0x0FF.
Task B looks at Address 0x0FF and it doesn't find "ABCDEF".

Reason:
0x0FF are two different Physical memory locations to the separate tasks. infact Task B might even have some code at its 0x0FF, or anthing!!!

thats what i said ... ok so I was more uncouth :-D

[bloodline]

AJCopland wrote:

HenryCase wrote:

AJCopland wrote:
Wow you can do that? (note: this is sarcasm)

Think of it like HD defragmentation if it helps you understand it.

Oh I understand what you're saying perfectly clearly. An old version of an arcade game conversion that I worked on used a shared pool of memory instead of doing full blown networking (don't 'cos I can't tell you which). Each machine was given a physical ID (using jumpers) and this told them what address they were allowed to write into, and conversely what addresses other machines on the "network" would be writing into.

This situation is similar to the AmigaOS how? You ask. Well with AOS you've got a linear address space. Which means that to program#1 a pointer with the address 0x0000FEED is the same as program#2s address of 0x0000FEED.

Now at this point my knowledge of workbench/AOS gets wonky 'cos i've never done any coding in it.

The above means that programs could, and probably did, pass around the pointer addresses to data rather than actual data itself because since nothing ever set or respected the memory protection flags it was more efficient to pass a 4byte memory address than a larger data structure like a string.

Implementing memory protection or memory management et al at this stage would break all of those existing programs, and probably workbench as well because they'd no longer be able to communicate with either the underlying OS or the other programs like they used too as each program would be within its own 4GB protected memory space and so the pointer addresses that it was merrily passing around would no longer match those of the program on the recieving end.

EDIT: The relevence to the first paragraph i wrote is that if those arcade machines OS had memory protection then they'd never get each others messages because whilst they'd all be accessing their _own_ versions of the correct memory addresses the physical section of memory that they'd be writing too would be different from the physical address that the other machines were reading from :EDIT

Now that might all be bollox because i've had a long day.

You are spot on. This is exactly the situation we are in with AmigaOS.

I also just realised that this was the ColdFIRE thread and I only came in here in the hope that someone would be commenting on THAT topic. So I apologise profusely to everyone for continuing this off-topic subject.

Andy

This is sort of related :-)

[bloodline]

Hans_ wrote:

bloodline wrote:

You put an OS layer inbetween memory allocation functions and the real memory. This MMU gives the memory allocation functions the impression that it is writing directly to the memory, when in reality it is controlling memory allocation/deallocation and protecting unwanted memory access.

I don't understand... The way memory protection would work is that each task gets it's own address space... in effect each task gets up to 4gig of address space to itself. I shall expalin the reprocussions of this later...

A good book on how MMU's work should give you a better understanding... The intel developer docs are a great place to start!

You don't need to create separate memory spaces to have full memory protection. In fact, now that many CPUs have 64-bit addressing, there are those who argue that a single linear address space is a good idea. Doing this would allow the existing messaging system to work fine, with the added restriction that programmers do need to mark the appropriate memory as shared.

If you have a flat VM... then you only have a single page table and you can't do per task protection!! Sure some pages can be read only... but really the only thing protected in that scheme would be the OS/Kernel... and that would give you a situation like Win95... shudder...

-Edit- Also, it would very hard to implement a VM paging to disk system with a flat address space... we all want  to be able to page memory to and from disk :-)

-Edit2- No, wait I see what you mean now! Separate spaces mapped to the same physical locations... yes that's doable, but the house keeping would be quite complex :-)

In order for Amiga OS to have full memory protection whilst still allowing old software to run, there is only one option: place the old apps in a sandbox. There the old apps can do all the naughty things that they used to get away with because the rules were never enforced. Unfortunately, this does mean significant rewrites of some OS modules such as intuition, which didn't follow the rules either.

Hans

This is very true. As apple did!!! Sandbox out the old apps.

[bloodline]

HenryCase wrote:

AJCopland wrote:
Wow you can do that? (note: this is sarcasm)

Think of it like HD defragmentation if it helps you understand it.

If you understand HD fragmentation... then you can apply that to how the MMU works... which is basically the hardware that allows the memory fragments (or pages) to be addressed as though they were ordinary RAM.

AJCopland wrote:
With all due respect Henry both bloodline and karlos are trying to explain how MMUs work from a position of knowing, at least from a code point-of-view, how they function. So some of your replies have read as being rather rude.

I'm not trying to be rude, but it's hard to maintain composure when you are being misunderstood. As I've just mentioned, lets call my idea an MAU (Memory Allocation Unit) as apposed to a MMU as I don't know if what I'm describing is your daddy's memory management system, if you catch my drift.

The problem is that the OS doesn't  and can't know the mind of the programmer... The Programmer, on AmigaOS, is free to do whatever he/she wants. Thus the OS can't control what the tasks do, and they can happily break all the rules (The Memory protection) that you might want to put into place, and they suffer no ill in AmigaOS... but enforce those rules and the tasks break.

[bloodline]

AJCopland wrote:

With all due respect Henry both bloodline and karlos are trying to explain how MMUs work from a position of knowing, at least from a code point-of-view, how they function. So some of your replies have read as being rather rude.

Andy

Henry is just passionate, I can understand that! If I can help steer his passion then I am happy to answer his questions!

OS Design is not an easy topic, it's good that he wants to think about it and does think about it!

If he is getting frustrated, then I blame my inability to explain the situation... I will endeavour to improve my communication skills!

-Edit- I think we can also blame the AmigaOS designers for cutting corners :-D

[Karlos]

HenryCase wrote:

Karlos wrote:
How do you take, say, 16 4KiB pages scattered across the 4G physical address space that an application requested and originally thought was one contiguous 64KiB lump of memory and tell it "here is where your data was" ?

By arranging it into a 64KiB lump before you give memory control back to the program.

Assuming you could do this, do you have any idea how complex the algorithm required to sort all the scattered physical blocks into contiguous lumps that reflect what the code originally allocated and ensuring pointers everywhere in the system are updated? That's not even including the overhead of copying pages of memory around.

Give me an example of when you'd use a pointer to address more than one memory location so I can explain how its done.

Any code that walks arrays, traverses containers, manipulates strings, etc.

[bloodline]

HenryCase wrote:

Kronos wrote:
@HenryCase

Problem is the OS can't decide wether one task made a legitemate request for data owned by another (lets say stuff running inside the input.device-task) or if it is just some roque pointer .....

I am not asking it to. I am asking the OS to make some limits on the commands that are physically not possible, like trying to put a 6-bit instruction into a 3-bit space. It was just one example of how the memory layer would keep the system stable.

But the OS can't limit anything, because it has no idea what the Tasks are passing to each other, or what they are accessing in the system...

Kronos wrote:
Next idea is usually to just copy system-structures into each tasks private part of memory, but once you take a look into the interlinked (and open) system-structures you'll notice that ain't pratical either.

Please expand on that point so I can understand it.

On a system like MacOX, to alter a system structure you have to use a specific OS function, this means that Apple can mess around with how the OS works and the Tasks are none the wiser... In AmigaOS due to it's open nature, tasks tended to alter system structures directly, which means the Amiga design team couldn't alter the system under hood... it also means they couldn't protect the OS anymore... as soon as you protect the OS, the tasks can't access the data they are used to accessing. Intuition, for example, actually documents this as an official method of working!!!! Apple would never allow anything like that!!!!

[Hans_]

bloodline wrote:

On a system like MacOX, to alter a system structure you have to use a specific OS function, this means that Apple can mess around with how the OS works and the Tasks are none the wiser... In AmigaOS due to it's open nature, tasks tended to alter system structures directly, which means the Amiga design team couldn't alter the system under hood... it also means they couldn't protect the OS anymore... as soon as you protect the OS, the tasks can't access the data they are used to accessing. Intuition, for example, actually documents this as an official method of working!!!! Apple would never allow anything like that!!!!

Way back when PowerPC cards were just coming out, programming guidelines were published that explicitly said something along the lines of:

In future OS structures will not be directly accessible. Use OS functions to access OS structures/data or your code will break in future versions of the OS.

The problem was, they never got round to providing all the accessor functions required.

Hans

[bloodline]

Hans_ wrote:

bloodline wrote:

On a system like MacOX, to alter a system structure you have to use a specific OS function, this means that Apple can mess around with how the OS works and the Tasks are none the wiser... In AmigaOS due to it's open nature, tasks tended to alter system structures directly, which means the Amiga design team couldn't alter the system under hood... it also means they couldn't protect the OS anymore... as soon as you protect the OS, the tasks can't access the data they are used to accessing. Intuition, for example, actually documents this as an official method of working!!!! Apple would never allow anything like that!!!!

Way back when PowerPC cards were just coming out, programming guidelines were published that explicitly said something along the lines of:

In future OS structures will not be directly accessible. Use OS functions to access OS structures/data or your code will break in future versions of the OS.

The problem was, they never got round to providing all the accessor functions required.

Hans

By the time the PPC cards were coming out, the OS had been out of development for 3 years... I think providing the accessor functions or not... that was far too little much much too late :-) Commodore cut corners with AmigaOS from the very beginning... to meet deadlines or keep performance up I don't know... but AOS2.0 really should have fixed an awful lot more than it did!

[bloodline]

@HenryCase

I think you'd probably find "Managed Code" operating systems rather interesting the most famous being MicroSoft's Singularity!

These put such controls on the programmer that no memory protection hardware is needed, this offers some advantages of AmigaOS without the fragile nature of AmigaOS's open memory model! Obviously it has disadvantages too... But still, some of the ideas are likely to make it into the mainstream!

[AeroMan]

I've never programmed a processor or microcontroller with MMU (unless you count PC code here, but I can't touch it), so let me ask to the MMU gurus here if my way of thinking is correct:

1: I suppose a task/process must be launched by the user or by the system at first instance.

2: The launched task may launch other daughter tasks if it needs.

Now the questions:

1: Am I wrong to suppose that the memory areas, will be shared only between the initial task, the system and the daughters ?

2: If so, why would somebody share memory with alien tasks for other purpouses than snooping ?

3: How often would software do this ? One per cent ? Ten ? One in a milllion ? :-)

4: Could we protect the memory in such a way that the task could only see what was allocated by itself and it's sub tasks ?

   This could work in a way like having the Amiga/AROS just for this task, and could avoid thrashing other tasks' memory.
   It looks fine for me, as it would ensure that a runaway task woud not mess with others.
 

[bloodline]

AeroMan wrote:
I've never programmed a processor or microcontroller with MMU (unless you count PC code here, but I can't touch it), so let me ask to the MMU gurus here if my way of thinking is correct:

1: I suppose a task/process must be launched by the user or by the system at first instance.

2: The launched task may launch other daughter tasks if it needs.

Now the questions:

1: Am I wrong to suppose that the memory areas, will be shared only between the initial task, the system and the daughters ?

Depends upon the OS, Windows threads exist very much in the Parent's address space... I think the same is true of Linux, though I'm sure they are more heavy weight than Windows threads.

On these two OSs I believe only threads share address space, programs don't, and they MUST communicate via carefully defined and controlled methods.

2: If so, why would somebody share memory with alien tasks for other purposes than snooping ?

AmigaOS doesn't have threads... everything is a task, if you added Memory Protection these tasks should not share address space... You would need to add threading library really (AROS has added some fields to the task structure to make them pthread compatible*) :-)  

*I would note that new functions would need to be added to exec or a new thread library to make use of them!

3: How often would software do this ? One per cent ? Ten ? One in a milllion ? :-)

We use Tasks as an example because it's easy to see what's going on... but all the arguments apply to Libraries, devices and in fact all system structures... The scale of the problem is huge!

Tasks and libraries are always poking around in structures they don't own.

4: Could we protect the memory in such a way that the task could only see what was allocated by itself and it's sub tasks ?

As stated above, that is only a tiny part of the problem.

   This could work in a way like having the Amiga/AROS just for this task, and could avoid thrashing other tasks' memory.
   It looks fine for me, as it would ensure that a runaway task woud not mess with others.
 

Memory Protection will break so much you wouldn't believe it! Blame Commodore :-)

[HenryCase]

@thread

First of all, I'd like to thank those that have responded to my memory protection posts, I've got a lot to digest there, and will do ASAP. However, it is late so I'll concentrate on bloodline's post quoted below, as it seems the most concise.

I would also like to state that I think memory protection would be useful with Coldfire, especially if we want to move away from emulating every 68k instruction.

@bloodline

bloodline wrote:

HenryCase wrote:
My challenge is this. I have a program running which has got an important string "ABCDEF" stored in memory at real location "0x001" and virtual location "0x0FF". Using your knowledge of the Amiga's API architecture, I want you to try and corrupt this data. I am confident that whatever you suggest can be dealt with without breaking AmigaOS compatibility, so try and prove me wrong.

Task A Stores "ABCDEF" at 0x0FF.
Task A sends a message to Task B.
The message simply contains 0x0FF.
Task B looks at Address 0x0FF and it doesn't find "ABCDEF".

Reason:
0x0FF are two different Physical memory locations to the separate tasks. infact Task B might even have some code at its 0x0FF, or anthing!!!

The OS has no idea that Physical address 0x0001 has to be mapped to the same place in both tasks... in fact it might not even be possible to map that physical address to the same virtual address!

Neither of the Tasks can ever know the Physical address, as their 0x0001 is mapped somewhere else... different for each task!!!

So if I understand correctly you're saying that the data cannot be read by more than one task because the OS doesn't know how to share this information?

If I have understood correctly, allow me to use your example  but fix it with the MAU (yes I am still going to use that term to avoid confusion):

Task A requests to store "ABCDEF", the MAU stores it at location 0x0001 and passes back a pointer to that location through assigning a 'virtual memory space' 0x0FF.
Task A sends a message through the MAU to Task B to look at memory space 0x0FF.
Task B makes the request to look at address 0x0FF, the MAU does not see a reason to not show this information, and fetches the string "ABCDEF" to pass on to Task B.

So in terms of memory usage, what do you have? You have one location where the string is stored in memory, a simple look up table stored in memory that is controlled by the MAU, the MAU itself which provides this abstraction layer and offers opportunities to control memory usage, and the memory spaces used by the tasks.

The address 0x0FF can be called as many times as you want (please note this virtual memory value can only be assigned to one real memory space at a time), but each time the MAU will make a decision based on certain rules to determine whether that access is given. It will also be giving programs non overlapping memory spaces to run in.

Does this explanation help?

EDIT: Should also say that I intend to separate kernel commands from the kernel itself so that its commands go through the MAU too (obviously with a high level of privileges). I thought that was worth pointing out in case you thought I was intending to do MP with the standard design Amiga kernel.