How to move AROS forward

[HenryCase]

@bloodline
I'll reply to your last post shortly, outlining my ideas for the new exec will take a little time.

@Hans

Hans_ wrote:
Anything running in supervisor mode should have full access to all memory in the system. Thus, the OS kernel wouldn't have to change ownership back and forth. For non-kernel OS modules that run in user mode, they could run directly in the application's task/process; this means that there still wouldn't be any ownership change.

Correct. The memory in use would be jointly owned by the kernel and the applications permitted to run in that space. This allocation only changes when you close an application or give a new application rights to use that space.

[bloodline]

HenryCase wrote:
@bloodline
Thanks for your response.

bloodline wrote:
What we have is a big flat area of memory, and a CPU.
The CPU does not know or care where it is reading and writing, it simply follows what it is told to do by a computer program. The CPU in AmigaOS has no information regarding the status of the code at all. While AmigaOS offers contexts (where each exec task has it's own set of certain data structures etc), these contexts are all part of the same address space.

I agree, the CPU doesn't have to care where it is reading or writing, it just has to follows what it is told to do by a computer program, namely the new AROS memory manager I am proposing. AROS is software too. I'm taking control over memory from the applications and giving it to the OS. If a program uses an API call like AllocMem() then AROS knows about it, right?

But in AmigaOS there is no way to control what a program does with that memory... It could quite happily pass the pointer returned by AllocMem() to another program... and that is a system legal thing to do in AmigaOS, this is a fundamental concept in AmigaOS that allow the system to function... if you have memory protection the pointer that the program received from the AllocMem() function doesn't exist anywhere but in that program's address space... so as soon as it tries to pass that pointer to another program/system function/etc it has just passed bad Data!! There is NO way in AmigaOS to know what a program is passing to another program.

bloodline wrote:
In the AmigaOS design, the designers decided that they wanted the system to be fast and use as little memory as possible. Thus AmigaOS was designed with the idea that any data structure that could be shared, would be shared... No provision was made in the design of Amiga to control the access to system wide data structures... this was a design decision, and it can't be retro fitted without totally redesigning the system, and it would no longer be AmigaOS compatible.

Clearly I'm not aiming to clone every design decision made by the AmigaOS designers.

Then you won't have AmigaOS... it would be like adding Gendered and Cased nouns to English... no existing English speaker would be able to use it... as it wouldn't be English anymore.

I'm trying to focus on how we keeping OS 3.1 API compatibility in a MP system, we can discuss if the resultant OS is Amiga-like later.

Amiga-like = AmigaOS 3.1 API compatibility. By the AROS definition.

bloodline wrote:
You are thinking high level, not machine level... Your ideas are actually an area of research called "managed code", the most famous is MicroSoft's Singularity project. There programs access memory via defined system functions thus memory access can be controlled without the need for an MMU.

Interesting. Can you give me an overview of how Microsoft's Singularity achieves this feat?

I already did... The concept of the memory pointer doesn't exist in a managed code system. All memory access is mediated by the Operating system. it's a slow way to operate, but it does allow stability on non MMU systems. The overhead of such a system make it unsuitable for real use... thus it was just a research project, now abandoned by MicroSoft, and released to the Open Source community.

Google and wikipedia are your friends here... I don't have to hold you hand the whole way?

bloodline wrote:
But AmigaOS is not a managed code system, it was designed as an operating system to be programmed at a machine code level... in fact I can write a program in ASM that would work on AmigaOS in a matter of minutes... I certainly couldn't even begin to think about doing that on a real operating like a UNIX or Windows... where I would have to stick with C.

AmigaOS may not have been a managed code system historically, but we're not looking at what it was but rather what we can make it without breaking compatibility.

what you are suggesting is the same as me saying... I shall try and cook a Chicken in a cement mixer... Just not possible... you would end up with mashed raw chicken...

You'd still be able to code in ASM with my MP system. The programs making AmigaOS API calls see the 'virtual' memory layout, but the users of the OS can see the 'virtual' memory layout and the 'real' memory layout. I hope you can see that gives us options to allow ASM code to run alongside other code.

Oops, I've confused you with my ASM example... I was simply trying to show how Simple the design of AmigaOS was... not that ASM was good or bad... One can easily write an computer program on Any OS in ASM (except a managed code system, were everything must be high level), but it's harder work because they are more complex.

One option is to add a new API call(s) especially for ASM code.

And what program is going to use a function call it doesn't know about!?!?

When the ASM program is executed it first makes an API call that says something like 'I will need memory from XXXX to XXXX to run'. If this memory is free then it is assigned to the ASM program. If it is not free you can either:

What? Why? This really makes no sense at all?!?

1. Close the programs using that memory.

Why... What if that program was your Email client?

2. Move the programs using the memory to another memory space (the programs are paused during this move, but can carry on afterwards as normal due to the 'virtual' memory addresses not moving.

How the hell do you move it? remember AmigaOS has no idea what program is using what memory...

You really must read about how virtual memory works as, this statement shows that you don't understand it at all!

3. A copy of the ASM program is made, and this copy is amended with available memory address locations before being run (the application doing this works like a find and replace function in a word processor).

Again, you haven't made any sense here...

However, ASM by its nature does not allow MP(AFAIK),

ASM has nothing to do with MP... that is a design consideration of the OS, not the programing language you have used.

unless you sandbox it, so yes you could crash AmigaOS with a badly coded ASM app, but the same is true of any operating system.

No it isn't... In a real OS... like OSX, no program can take down the OS... no matter how you wrote it. that is the point of MP.

If you code ASM apps so that they stay out of the way of the area where the OS is running from you should be okay.

That is the point of MP... so if a program tries to hit the OS space... it can't, this is done by sitting a program in it's own address space, where the OS does not exist... the App has no ability to see any other memory than what is in it's own address space... once the CPU is in that task's context it cannot access (read or write) outside of that context's address space.

[bloodline]

HenryCase wrote:
@bloodline
I'll reply to your last post shortly, outlining my ideas for the new exec will take a little time.

@Hans

Hans_ wrote:
Anything running in supervisor mode should have full access to all memory in the system. Thus, the OS kernel wouldn't have to change ownership back and forth. For non-kernel OS modules that run in user mode, they could run directly in the application's task/process; this means that there still wouldn't be any ownership change.

Correct. The memory in use would be jointly owned by the kernel and the applications permitted to run in that space. This allocation only changes when you close an application or give a new application rights to use that space.

No, Hans_ is confusing the issue.

If you allow a system structure to be shared between apps, then it only takes one app to to accidentally or maliciously destroy that data structure to take that component down and the system is now broken...

But shared memory areas are not a problem, and most OSs do allow a shared memory space, but their use is very carefully defined in the OS design, and nothing mission critical can be put there.
 
With AmigaOS, you have NO way to monitor what is being passed between programs/system functions... this is a design feature of AmigaOS... it is defined by the way AmigaOS works!

As soon as you add memory protection, the programs and functions will be passing rubbish between each other... and there is nothing you can do about that, because of the design of AmigaOS!!!

If you add MP, you will be altering so much of the system that you will end up with a new operating system that is totally incompatible with AmigaOS 3.1... and if you are going to that much trouble, why not just take an existing system that already has the design features you want...

In 1998 Apple had the same problem, MacOS was not able to have MP or multitasking by design... 6 years of attempting to retrofit them was fruitless... so they brought in a new system and then put an Emulator to run the old Apps... that would have been the only solution for AmigaOS too.

[paolone]

Paolone, I don't know if I was meant to be one of the targets of your 'all talk no action' rant or not

Nope. And I've found your proposals for a PM scheme very interesting: at least you're proposing a solution, instead of simply whining 'cos there's no PM in AROS. My rants were almost directed to the initial post.

Thanks for your vision on VmwAROS, I'm really happy it has been well received by the community. But VmwAROS is in first place an example of my "take action" motto: I felt AROS needed a modern, complete distribution since AROS-MAX was dead, and I simply MADE it.

I'm neither a C developer nor a IT graduate, I'm simply what someone would call "power user". VmwAROS demonstrates that everyone of us, instead of simply whining or struggling in a stupid poor men's war, has the strenght to actually do something. It requires just a lot of will and some spare time.

[SamuraiCrow]

Hans_ wrote:
@SamuraiCrow

I think the answer to the issues that are often associated with memory protection are generally better addressed by using "safe" programming languages like Oberon and some versions of Basic that either don't directly support pointers in general, or have tight limitations on the use of arrays and other data structures to monitor them for overruns and NULL pointer values internally.

The problem with that is that it requires all software to play by the rules. Malicious code does not. Memory protection helps prevent malicious code from doing harm by restricting access to resources.

Hans

Memory protection doesn't stop malicious code either.  In a black-box operating system malicious code can overwrite anything that is in the same memory page as it is.  In a white-box operating system it can corrupt any system that the code has access to.

The problem with memory protection is that it is limited to memory granularity of the page size of the MMU which is usually 4k.  If the assign command is only a couple hundred bytes long and you add memory protection to it it becomes 4k for the code, 4k for the stack, and 4k for any global data.  That's 12k for the smallest program to run.  That eliminates all of the streamlined efficiency of the AmigaOS when it needs more memory than it allocates just to run a small command.

[Hans_]

SamuraiCrow wrote:

Hans_ wrote:
@SamuraiCrow

I think the answer to the issues that are often associated with memory protection are generally better addressed by using "safe" programming languages like Oberon and some versions of Basic that either don't directly support pointers in general, or have tight limitations on the use of arrays and other data structures to monitor them for overruns and NULL pointer values internally.

The problem with that is that it requires all software to play by the rules. Malicious code does not. Memory protection helps prevent malicious code from doing harm by restricting access to resources.

Hans

Memory protection doesn't stop malicious code either.  In a black-box operating system malicious code can overwrite anything that is in the same memory page as it is.  In a white-box operating system it can corrupt any system that the code has access to.

It's no panacea, but it does restrict the amount of damage that can be done. Private memory belonging to multiple processes shouldn't be on the same page. Using more memory than necessary for tiny programs is simply a price that you have to pay.

Let me put it this way, while programming languages/compilers could theoretically generate safe code, there's nothing to stop someone from deliberately generating nasty code. Being able to compromise the whole system with a few trivial instructions is not acceptable. Sure, you may say, just don't run code from people you don't trust. Ok, how do you know who to trust? Protecting the system is part of a modern OS' job.

Hans

[HenryCase]

@bloodline
Whilst thinking about how we would need to change the core of AmigaOS to allow for MP, I thought of a talking point we can use to (hopefully) progress our discussion into something more productive. Am I right in thinking that your main argument behind 'no MP' with the OS3.1 API is because you can't think of a way of adding resource tracking without breaking compatibility?

If I am correct, then lets look at how API calls with MP and non-MP operating systems compare. Lets take the allocation of memory API call for example. In AmigaOS we have AllocMem:

http://utilitybase.com/ref/?keyword=AllocMem&funcgroup=AmigaOS&action=Search
"memoryBlock = AllocMem(byteSize, attributes)"

Now let's look at malloc, the equivalent memory allocation function in FreeBSD, an OS with MP:
http://www.freebsd.org/cgi/man.cgi?query=malloc&sektion=9
"malloc(unsigned long size, struct malloc_type *type, int flags);"

Okay, so let's compare these API calls. First we can see they both have attributes that allow you to set the size of memory space required. They also both have flags/attributes you can set to determine the behaviour of that memory space. With AllocMem (AmigaOS), we have the following flags:

MEMF_CHIP (request ChipMem), MEMF_FAST (request FastMem), MEMF_LOCAL (memory survives CPU reset), MEMF_24BITDMA (used for setting memory for Zorro devices), MEMF_KICK (allows memory to be accessed by Exec early in boot process), MEMF_CLEAR (sets memory to all 'zeros'), MEMF_REVERSE (allocate from top of memory pool), MEMF_NO_EXPUNGE (prevents memory from being expunged on failed memory allocation)

Now let's look at the flags we can set using malloc (FreeBSD):

M_ZERO (sets memory to all 'zeros'), M_NOWAIT (returns value if memory cannot be set immediately), M_WAITOK (indicates okay to wait for memory to be set), M_USE_RESERVE (indicates memory can be set in memory reserved for OS use)

Take a good look at AllocMem and malloc. If I didn't tell you that one function was used on a non-MP OS and the other was on a MP OS, would you have been able to tell just by looking at the API call structure? No.

Similarly, the API call itself doesn't care if it's running on a MP OS or non-MP OS. There is no 'anti-MP' flag for AllocMem. Resource tracking is a separate issue from API calls, it is what the kernel handles.

Thoughts?

[bloodline]

HenryCase wrote:
@bloodline
Whilst thinking about how we would need to change the core of AmigaOS to allow for MP, I thought of a talking point we can use to (hopefully) progress our discussion into something more productive. Am I right in thinking that your main argument behind 'no MP' with the OS3.1 API is because you can't think of a way of adding resource tracking without breaking compatibility?

No, not at all. AROS in fact implements resource tracking.

If I am correct, then lets look at how API calls with MP and non-MP operating systems compare. Lets take the allocation of memory API call for example. In AmigaOS we have AllocMem:

http://utilitybase.com/ref/?keyword=AllocMem&funcgroup=AmigaOS&action=Search
"memoryBlock = AllocMem(byteSize, attributes)"

Now let's look at malloc, the equivalent memory allocation function in FreeBSD, an OS with MP:
http://www.freebsd.org/cgi/man.cgi?query=malloc&sektion=9
"malloc(unsigned long size, struct malloc_type *type, int flags);"

Okay, so let's compare these API calls. First we can see they both have attributes that allow you to set the size of memory space required. They also both have flags/attributes you can set to determine the behaviour of that memory space. With AllocMem (AmigaOS), we have the following flags:

MEMF_CHIP (request ChipMem), MEMF_FAST (request FastMem), MEMF_LOCAL (memory survives CPU reset), MEMF_24BITDMA (used for setting memory for Zorro devices), MEMF_KICK (allows memory to be accessed by Exec early in boot process), MEMF_CLEAR (sets memory to all 'zeros'), MEMF_REVERSE (allocate from top of memory pool), MEMF_NO_EXPUNGE (prevents memory from being expunged on failed memory allocation)

Now let's look at the flags we can set using malloc (FreeBSD):

M_ZERO (sets memory to all 'zeros'), M_NOWAIT (returns value if memory cannot be set immediately), M_WAITOK (indicates okay to wait for memory to be set), M_USE_RESERVE (indicates memory can be set in memory reserved for OS use)

Take a good look at AllocMem and malloc. If I didn't tell you that one function was used on a non-MP OS and the other was on a MP OS, would you have been able to tell just by looking at the API call structure? No.

Similarly, the API call itself doesn't care if it's running on a MP OS or non-MP OS. There is no 'anti-MP' flag for AllocMem. Resource tracking is a separate issue from API calls, it is what the kernel handles.

Thoughts?

Ok, I can see where you are missing the problem I think.

The problem is not that you can't add MP to AmigaOS. The problem is that when you add it, the system doesn't work like AmigaOS any more.

In an MP OS, you pass data between programs and system functions. This is how they communicate. This is clearly defined behavior, as the OS and the programs "know" that they all live in separate address spaces... which are to all intents and purposes separate computers.

In AmigaOS, programs pass pointers, NOT DATA, to each other... these pointers reference memory anywhere in the flat address space.  As soon as each task gets it's own address space, the pointers which are passed between programs are meaningless!!! Thus all amiga software stops working! There is no way to fix this since AmigaOS only has one message passing system and the OS puts no controls on what is passed.

[bloodline]

As an example,

Two task in AmigaOS... A and B

Task A will allocate some memory and fill it with some Data, and then send a pointer to Task B.  Task B receives that pointer and uses it to access the Data put there by Task A. This is a very quick process and requires only the memory allocated by Task A.

In an MP OS there are several strategies used to pass data, I will only present a simple one here.

Task A will allocate some memory and fill it with Data. It then must actually send that Data to task B, this means physically copying the Data from Task A's address space to Task B's Address space byte by byte...
there are actually other strategies that one might employ (since this is rather slow) but it shows clearly the problem which we have to overcome!
Task B must have it's own copy of the same Data in it's own address space in order to work on it.

Amiga OS can happily use the slower more complex strategies for passing data, which are MP compatible... but these methods were never required and thus not used either by the OS or any applications... so we are stuck with a non MP compatible OS... since there is no way to know what the OS and task are passing to each other!

[bloodline]

In your AllocMem() Vs malloc() example...

The difference is this;

AllocMem() returns an address that points to the same bit of memory to any program that were to receive that pointer.

malloc() returns an address that is meaningful only to the task that called the function... any other task that receives that pointer has just received a meaningless pointer which points to nowhere that it can address!

[Leo24]

Hi guys,

sorry I didn`t read the whole thread...but here some notes from me. I suppose MP in this context means Multiprocessing, right?
It`s true that most comercial and free operating systems out there use a separate address space for each process. This is a design decision** and not mandatory nor necessary for a MP capable OS. It`s possible to have MP working on a single address space and to use the MMU to extend this space beyond the physical memory by using the page swaping techique and to handle memory protection. Shared resources must still be protected through mutexes or semaphores from concurrent CPU accesses. In a single (flat) address space OS (SASOS) message passing is done by passing pointers to the data in memory, just like AmigaOS does, there is no need to move or copy data around. Note that even in non-SASOS data is generally not copied around, it`s mapped with the help of the MMU, so both processes may see the same data in the same place or in case a process already allocated that virtual memory region for other stuff, in different memory places.

** in the early days of computing when there was 16 bit address spaces only, address translations didn`t make much sense because one could not get over the 16 bit address space with it. Earlier MMUs just swapped memory regions with the bank switching method, thus having many code blocks using the same adresses but only one at a time. In the 32 bit era this technique became obsolete and segmenting and paging took over. However, using for each process different address spaces mapped to the same region simplified program loading as programs could be linked statically and did not need to be relocated in memory. In the 64 bit era there is realy no need to have many address spaces any more. Newer research OSes (e.g. singularity from Microsoft) are in fact SASOS.

Best regards,
Leo24

[bloodline]

Leo24 wrote:
Hi guys,

sorry I didn`t read the whole thread...but here some notes from me. I suppose MP in this context means Multiprocessing, right?

No, we are talking about Memory Protection.

Lets not even go near Multiprocessing yet as that is a whole other issue, which AmigaOS isn't well suited to...

It`s true that most comercial and free operating systems out there use a separate address space for each process. This is a design decision** and not mandatory nor necessary for a MP capable OS. It`s possible to have MP working on a single address space and to use the MMU to extend this space beyond the physical memory by using the page swaping techique and to handle memory protection. Shared resources must still be protected through mutexes or semaphores from concurrent CPU accesses. In a single (flat) address space OS (SASOS) message passing is done by passing pointers to the data in memory, just like AmigaOS does, there is no need to move or copy data around. Note that even in non-SASOS data is generally not copied around, it`s mapped with the help of the MMU, so both processes may see the same data in the same place or in case a process already allocated that virtual memory region for other stuff, in different memory places.

AmigaOS tasks often rely on a Forbid() halting all multitasking and that a high priority task WILL block a lower priority one... on a Multiprocessor system this condition may not always be true.

** in the early days of computing when there was 16 bit address spaces only, address translations didn`t make much sense because one could not get over the 16 bit address space with it. Earlier MMUs just swapped memory regions with the bank switching method, thus having many code blocks using the same adresses but only one at a time. In the 32 bit era this technique became obsolete and segmenting and paging took over. However, using for each process different address spaces mapped to the same region simplified program loading as programs could be linked statically and did not need to be relocated in memory. In the 64 bit era there is realy no need to have many address spaces any more. Newer research OSes (e.g. singularity from Microsoft) are in fact SASOS.

Best regards,
Leo24

[itix]

@HenryCase

an you talk me through the way OpenWindow() is used in OS 3.1 so that I can look at creating this separation based on how it actually works and not my basic interpretation of Amiga API guides from the Internet.

Uhm... You can not learn it from API guides. Essentially because many actions are not done through API. For example windows can have UserPort. This is allocated by operating system or by application and sometimes there is no UserPort at all. Sometimes application creates its own UserPorts and pokes this into struct Window and Intuition starts using it when you call ModifyIDCMP(). To make things even more complex it is perfectly legal that application A opens a window, application B supplies UserPort to that window and application C manages this window (maybe drawing something). And then there is Intuition task called input.device which manages all windows. You can not assume anything about ownership of that window and its associated structures.

Similarly you can not know who is really going to use memory allocated by AllocMem() or malloc() (in the end malloc() calls AllocMem()). On Amiga you can share all memory allocated by AllocMem()/malloc(), you can even share your code and structures allocated from the stack. There is no absolutely privacy on Amiga. Everything you do is exposed to other applications and both applications and kernel exploit this.

Especially Kickstart 2.0 added many aspects to API which make adding MP even more difficult than it was for Kickstart 1.3.

I should also remind you that OpenWindow() is only one of those few hundred functions which should change to introduce memory protection. Even message passing system in Amiga is flawed in this sense.

To put it short Amiga API is completely broken and can not get fixed. If you want MP, that is

[itix]

@bloodline

No, not at all. AROS in fact implements resource tracking.

Does it also work? :-)

[bloodline]

itix wrote:
@bloodline

No, not at all. AROS in fact implements resource tracking.

Does it also work? :-)

Not really... :-D

[HenryCase]

@bloodline
I'm glad we're getting somewhere now, you were right I was unaware of the pointer usage in AmigaOS.

bloodline wrote:
As an example,

Two task in AmigaOS... A and B

Task A will allocate some memory and fill it with some Data, and then send a pointer to Task B.  Task B receives that pointer and uses it to access the Data put there by Task A. This is a very quick process and requires only the memory allocated by Task A.

Then what we would need for MP would be a way of sticking the OS in-between all communications between tasks by manipulating the pointer data. First we need a way of flagging up when a pointer has been set by a program. I've used this guide to give me an overview of how pointers work:
http://www.codeproject.com/KB/cpp/pointers.aspx
So in C you set the pointer variable with a command like this:
int* pNumberOne; (for a pointer of integer type)

Now going to compare this with Pascal. Using this guide...
http://www.learn-programming.za.net/programming_pascal_learn13.html
So if you wanted to declare the same pointer in Pascal you would use these commands:
var
   pNumberOne: ^integer;

My question is this. When it comes to compiling the C and Pascal programs will they have the same machine code value for the pointer we declared?