How to move AROS forward

[HenryCase]

Matt_H wrote:
With the current lack of manpower, formally forking AROS will just create 2 stagnant projects.

Bingo.

saimon69 wrote:
Inside the same thread a proposal from HenryCase in order to avoid a fork and provide both new ways to evolve AROS (such as the holy memory protection grail) and in the same time provide the compatibility with the 3.1 API, but cannot say more about it because he haven't laid out anywhere yet, well maybe this is the right time....

Well remembered. I'm not going to say any more on the subject until I'm ready to code a 'proof' for my theory (the MP issue always causes a sh*tstorm), so you're right, now is not the time.

uncharted wrote:
Proposing a sensible path forward (see above - you did actually read it all before going onto the defensive didn't you?)

uncharted, thank you for ideas, but as you can see from saimon69's post AROS fans already discussed these same ideas a few months ago. Some would argue we lost a good developer out of those discussions (though there were other factors at play too).

I'd say yakumo9275's response was justified. Too many armchair experts (including myself), not enough developers, that's AROS's main problem, so what are you going to do to fix that?

Anyway, contribute something concrete to AROS and I'm sure you'll get a more favourable response to your ideas.

[HenryCase]

yakumo9275 wrote:
You really need to explain your classic idea and sell it coz I dont see the point of it. If genuine 3.1 wasnt available, then I could maybe see a point to it.

I know you were talking to uncharted here, but what the hell, let's open this can of worms again...

68k AROS + new 68k Kickstart = freedom for classic machines, giving a much more expandable and customisable OS. Useful for the machines we owe so much to as well as the new 68k Amigas (Minimig, Natami, etc...). To turn your question around, why would you not want that?

[HenryCase]

@bloodline

bloodline wrote:
Hahaha, but no... Go on fork it now, it's easy! Download the souce code, set up an SVN server, and then develop it how ever you like... Try and get devs on board though and see how far you get... If you make a sucess then what have we lost?

I agree that creating a fork isn't necessarily hard (or a bad thing for that matter) but I would prefer to keep the number of forks to a minimum. Where do you think I would be getting those developers from, the AROS dev community or elsewhere? If I'm going to get new devs involved, I'd much rather they worked on the main x86 (or x86-64) AROS branch.

bloodline wrote:
But like most in the Amiga scene it's all just talk... at least the AROS team as it is DOES achieve something.

I agree, and that's why I admire the AROS devs, they're making a real positive difference in the Amiga community.

bloodline wrote:
I summise that the current AROS devs just work in their spare time and work on what parts they understand or find interesting. They do a fine job. If you make fork, they're not going to work on that fork (they don't have the time or the motivation)... the Main AROS tree wil develop as it always has, and your fork will die.

The only (new) fork I would be interested in would be a native 68k port. There are other projects I would rather complete first as I don't possess all the necessary skills to work on a 68k fork. The first major project I'd like to tackle is an AROS audio player, though I must admit that I'm still in the early stages of learning C programming properly.

bloodline wrote:
It can't be done... the AmigaOS design does not allow for MP... I am happy to talk you through it again... it's a topic I actually enjoy talking about.

Glad you feel that way bloodline, I didn't want to feel like there was a negative vibe after our last discussions on the subject (and other subjects), but I did feel that vibe, which is one of the reasons I held off.

So lets start this discussion again, but first a question... can you name me two Amiga programs that are designed to trade data with each other through memory addresses?

bloodline wrote:
Yes, we lost a great developer... but even though he has now left, he contributed a great deal to the AROS project!

Yes, Robert N did contribute a great deal to AROS, but to those who are unaware of the situation it's best we clarify the outcome of the 'AROS fork' discussion he started was not the main reason he left us.

bloodline wrote:
I don't think AROS actually has a problem at all... we are half way throguh 2008... the whole concept of the Amiga is VERY VERY dead.

AROS is now very much a hobby project, and as such it's actually doing very well. I can install it on my various computers, it functions just as my old Amigas function, it's fun to play with... it costs me nothing to mess around with... I have the source code if I want to try out an operating system idea, I can try it out, because of AROS I understand how AmigaOS actually works and the design choices made and why... and seeing the flaws in it made me apprciate the design choices and trade offs made in other operating systems...

AROS doesn't have a problem in the sense that it is still alive and progressing, but it has great potential that is still untapped, which we would be able to tap with more developers. In my opinion AROS should be the most popular of the three modern Amiga based OS's, why do you think that it is not?

[HenryCase]

@bloodline
Don't worry about answering my previous question, let's start the MP discussion now.

Okay, I'd like to start this post by making a couple of things clear. First of all, there are three separate questions we could discuss (as far as I can see) when it comes to the possibility of MP:

1. Can it be done?
2. Can it be done elegantly?
3. Should it be done?

I hope you will agree that there is little point answering questions 2 and 3 if we do not agree that the answer to question 1 is yes, so that's what we should focus our discussion on IMO (at least at first). What this means is that if the MP implementation is a complete bodge job but works successfully then it can satisfy the criteria for answering question 1 and we can move on to the 'should we/how best do we' do it part of the discussion.

Next, I'd like to outline what I'd class as successful MP. To me the job of MP isn't to make individual applications less prone to crashing, but rather to ensure that when applications crash they don't take down the operating system and other applications with them. That is the base functionality of MP IMO, do you agree? I would also say that close to that core MP functionality is the ability for the OS to attempt to recover the data the application was using before the crash, but that is not a function I will focus on for now.

==============================================================
Right, down to business. A program in AROS is loaded. The memory manager assigns a unique ID to that program. The program starts requesting RAM space from AROS to store its data. These calls will either be for:

A space in memory of a specific size but undefined location.
or...
A space in memory of a specific size and location.

The AROS memory manager I am proposing doesn't give programs a memory address based on the location requested, but rather provides a 'virtualization' layer. When the program then tries to call data from a memory address it thinks is set in RAM, the memory manager calls the data from where it is really stored and presents it to the program. The program doesn't know the difference.

Lets take an example to illustrate this further. The program Lunapaint is loaded. AROS assigns Lunapaint the ID 0111. Lunapaint then requests a 5-bit memory address space starting at address 3FDF. That address is already taken in RAM by the program Wookiechat, but this doesn't matter to us. The 5-bit memory space is put in the address starting A21C instead. When Lunapaint tries to read or write to the memory space it thought was at 3FDF, the memory manager knows that the data stored in 3FDF for program ID 0111 is actually at A21C and pulls this data instead.

So what does this mean? It means the OS has taken control of memory allocation away from programs without letting the programs know there is anything different happening. Now lets look at what happens when programs try to pass data between each other through memory addresses.

What we want to be able to define here are the processes that should be allowed to share memory addresses with each other, and those programs that should not be allowed to share memory addresses (prevent problems like buffer overflows for example). The approach I'm proposing here is what I'd call the 'AI' approach, even though the intelligence is not artificial... I propose we use our own brains!

When two programs try to write to the same address space, we get a dialog box pop up on AROS (much like an Internet firewall, such as ZoneAlarm) asking whether we would like for these programs to be able to share memory address space. Taking the Lunapaint/Wookiechat example used above, the dialog box would ask us if Lunapaint and Wookiechat should share address spaces. Our options could be something like:

Share 3FDF, no universal share.
Do not allow programs to share any memory address spaces.
Allow both programs full access to each others memory address spaces.

You might think this would be a pain to set up (having a dialog box pop up every time you open two certain programs at the same time), but a way around this is to use patches. How would this work? Well for patches to be applied the memory structure needs to be consistent every time the program is run (so that if I apply a patch setting to 3FDF one day the patch will work on the same data the next day). However, as we have 'virtualized' memory allocation this is possible. The memory manager is the OS layer doing all of the assigning of memory locations. When a program asks for a memory address in a specific location, we give it a memory location with that label. When a program asks for memory but doesn't specify where that memory should be we make sure that the 'virtualized' memory allocation follows rules, so that we will always be able to find that memory.

For example, on program boot up we are dealing with requests for memory that are always in the same order, so why not give them the labels 0001, 0002, 0003, 0004, etc... From this point on the memory needs of the application are dynamic, so we look at assigning memory labels not on sequence but on a mathematical algorithm based on how they are first assigned and the memory spaces already in use (there may be an easier way of doing this, basically I'm looking for ways to pinpoint the setting of a variable in a program).

Assuming the above variable pinpointing is possible, then we can apply our patches and they can work consistently every time. The memory labels assigned don't affect real memory usage in the sense that all programs can have the 0001, 0002, 0003, etc... labels and the real memory can still be fully utilised.

What would we use these patches for? Well, let's say we never want Lunapaint and Wookiechat to ever share memory addresses. We can create a patch for this. We can also create a patch so that Lunapaint will share certain memory addresses with Wookiechat but won't share those same memory addresses with UAE. The beauty of the patch system is you can make your system as secure (or as open) as you like. You could also distribute these patches between AROS users (like  the way WHDLoad patches are created and distributed).

Okay, so we've looked at taking control of memory allocation, and how we can let programs interoperate in memory with our permission, the final piece of the puzzle is how we handle application crashes. Let's say I write a program for AROS, but I make a mistake so that in certain circumstances I'm trying to write a long string into a space defined for an integer, which causes the program to freeze/crash due to a buffer overflow.

Where does this buffer overflow go? How do we know the program is misbehaving? We have options here. We could let the program write the extra bits into a new 'virtual' address, hoping that the program will run smoothly afterwards (and have rules about how often this 'overflow' storage can happen before we flag up an error to the user) or we could simply flag an error as soon as the the buffer overflow happened. Link this with CPU throttling (so that programs caught in an infinite loop can have their CPU time reduced to avoid crashes) and you have a system that allows you to stop any misbehaving programs.
==============================================================

So we've looked at memory allocation control, memory permissions and crash control. Is there any other factors an OS needs for MP?

Note that the only part in my mind with a question mark over it is how to flag individual variables for 'patching', I'll have to look at how other systems (like compilers) do this, but even if this is not possible you'd still be able to have interoperability settings at the application level (i.e. Program 1 can share memory addresses with Program 2 but not with Program 3), which is still useful for system stability.

Please let me know what you think bloodline. Thanks.

[HenryCase]

koaftder wrote:
At some point, most people eventually learn when the fundamental architecture of a system is no longer worth saving...

That falls into the 'should it be done' MP question category. Do you have any input on the 'can it be done' MP question (based on my last post in this thread)?

[HenryCase]

SamuraiCrow wrote:
Regarding memory protection:

1. Can it be done?
2. Can it be done elegantly?
3. Should it be done?

1.  It can only be done part way and it requires an MMU.
2.  No, managed code is much more elegant even if it is sometimes a bit slow.
3.  No.  If we are going to support the Natami and MiniMig as a starting place, we're going to have to realize that the fact that the 68070 on the production Natami and 68000 on the MiniMig will not have an MMU is going to prevent a memory-protected AROS from running at all!

SamuraiCrow thanks for your reply, but you're jumping ahead in the discussions here. As I said before I see no point discussing questions 2 and 3 until we come to an agreement on the answer to question 1. The only way you're going to convince me that MP can only be done partially is to find flaws in the MP solution I outlined earlier in this thread. Please do so. Thanks in advance.

[HenryCase]

itix wrote:
@HenryCase
One problem in memory protection is that applications and the operating system are sharing system structures. For example when you call OpenWindow() the system returns struct Window which is globally shared. If your application trashes this structure it may lead to crash of operating system.

Thanks for your reply itix. I looked into the OpenWindow() API call and I can't see anything that would prevent changing it from a global variable to a more local one.

itix wrote:
The design of Amiga is unfortunately flawed for memory protection. One could say that on Amiga there is no user space and kernel space -- there is only kernel space where all applications are running.

This is the case now, but we're looking at ways to create a user space-kernel space separation whilst keeping OS3.1 API compatibility. This is not your daddy's exec if you catch my drift. If it is possible to implement a new kernel with MP that supports the OS3.1 API calls then we can have MP in an Amiga based OS.

So a good question is this: are there any global variables that cannot be contained more locally?

[HenryCase]

paolone wrote:
The direction is already here for the people that want to see it. The others may disagree, but please don't claim AROS is pointless or directionless anymore, as it is completely untrue.

Paolone, I don't know if I was meant to be one of the targets of your 'all talk no action' rant or not, but I wanted to tell you that I really believe VmwAROS is a great asset to the AROS (and Amiga) community, and is helping to bring some structure back to AROS application development. Thank you for your continued work on it.

[HenryCase]

itix wrote:
Local in which way? Intuition reads and writes to this structure and application reads and writes to this structure. There is no way you could make it "local" and retain API compatibility.

Can you talk me through the way OpenWindow() is used in OS 3.1 so that I can look at creating this separation based on how it actually works and not my basic interpretation of Amiga API guides from the Internet.

itix wrote:

If it is possible to implement a new kernel with MP that supports the OS3.1 API calls then we can have MP in an Amiga based OS.

If it was possible AROS would have implemented it already.

I don't believe that MP was/is inevitable. There were more important tasks to complete when the AROS kernel was designed and created, like just getting it to work.

[HenryCase]

@bloodline
Thanks for your response.

bloodline wrote:
What we have is a big flat area of memory, and a CPU.
The CPU does not know or care where it is reading and writing, it simply follows what it is told to do by a computer program. The CPU in AmigaOS has no information regarding the status of the code at all. While AmigaOS offers contexts (where each exec task has it's own set of certain data structures etc), these contexts are all part of the same address space.

I agree, the CPU doesn't have to care where it is reading or writing, it just has to follows what it is told to do by a computer program, namely the new AROS memory manager I am proposing. AROS is software too. I'm taking control over memory from the applications and giving it to the OS. If a program uses an API call like AllocMem() then AROS knows about it, right?

bloodline wrote:
In the AmigaOS design, the designers decided that they wanted the system to be fast and use as little memory as possible. Thus AmigaOS was designed with the idea that any data structure that could be shared, would be shared... No provision was made in the design of Amiga to control the access to system wide data structures... this was a design decision, and it can't be retro fitted without totally redesigning the system, and it would no longer be AmigaOS compatible.

Clearly I'm not aiming to clone every design decision made by the AmigaOS designers. I'm trying to focus on how we keeping OS 3.1 API compatibility in a MP system, we can discuss if the resultant OS is Amiga-like later.

bloodline wrote:
You are thinking high level, not machine level... Your ideas are actually an area of research called "managed code", the most famous is MicroSoft's Singularity project. There programs access memory via defined system functions thus memory access can be controlled without the need for an MMU.

Interesting. Can you give me an overview of how Microsoft's Singularity achieves this feat?

bloodline wrote:
But AmigaOS is not a managed code system, it was designed as an operating system to be programmed at a machine code level... in fact I can write a program in ASM that would work on AmigaOS in a matter of minutes... I certainly couldn't even begin to think about doing that on a real operating like a UNIX or Windows... where I would have to stick with C.

AmigaOS may not have been a managed code system historically, but we're not looking at what it was but rather what we can make it without breaking compatibility. You'd still be able to code in ASM with my MP system. The programs making AmigaOS API calls see the 'virtual' memory layout, but the users of the OS can see the 'virtual' memory layout and the 'real' memory layout. I hope you can see that gives us options to allow ASM code to run alongside other code.

One option is to add a new API call(s) especially for ASM code. When the ASM program is executed it first makes an API call that says something like 'I will need memory from XXXX to XXXX to run'. If this memory is free then it is assigned to the ASM program. If it is not free you can either:

1. Close the programs using that memory.
2. Move the programs using the memory to another memory space (the programs are paused during this move, but can carry on afterwards as normal due to the 'virtual' memory addresses not moving.
3. A copy of the ASM program is made, and this copy is amended with available memory address locations before being run (the application doing this works like a find and replace function in a word processor).

However, ASM by its nature does not allow MP(AFAIK), unless you sandbox it, so yes you could crash AmigaOS with a badly coded ASM app, but the same is true of any operating system. If you code ASM apps so that they stay out of the way of the area where the OS is running from you should be okay.

[HenryCase]

@bloodline
I'll reply to your last post shortly, outlining my ideas for the new exec will take a little time.

@Hans

Hans_ wrote:
Anything running in supervisor mode should have full access to all memory in the system. Thus, the OS kernel wouldn't have to change ownership back and forth. For non-kernel OS modules that run in user mode, they could run directly in the application's task/process; this means that there still wouldn't be any ownership change.

Correct. The memory in use would be jointly owned by the kernel and the applications permitted to run in that space. This allocation only changes when you close an application or give a new application rights to use that space.

[HenryCase]

@bloodline
Whilst thinking about how we would need to change the core of AmigaOS to allow for MP, I thought of a talking point we can use to (hopefully) progress our discussion into something more productive. Am I right in thinking that your main argument behind 'no MP' with the OS3.1 API is because you can't think of a way of adding resource tracking without breaking compatibility?

If I am correct, then lets look at how API calls with MP and non-MP operating systems compare. Lets take the allocation of memory API call for example. In AmigaOS we have AllocMem:

http://utilitybase.com/ref/?keyword=AllocMem&funcgroup=AmigaOS&action=Search
"memoryBlock = AllocMem(byteSize, attributes)"

Now let's look at malloc, the equivalent memory allocation function in FreeBSD, an OS with MP:
http://www.freebsd.org/cgi/man.cgi?query=malloc&sektion=9
"malloc(unsigned long size, struct malloc_type *type, int flags);"

Okay, so let's compare these API calls. First we can see they both have attributes that allow you to set the size of memory space required. They also both have flags/attributes you can set to determine the behaviour of that memory space. With AllocMem (AmigaOS), we have the following flags:

MEMF_CHIP (request ChipMem), MEMF_FAST (request FastMem), MEMF_LOCAL (memory survives CPU reset), MEMF_24BITDMA (used for setting memory for Zorro devices), MEMF_KICK (allows memory to be accessed by Exec early in boot process), MEMF_CLEAR (sets memory to all 'zeros'), MEMF_REVERSE (allocate from top of memory pool), MEMF_NO_EXPUNGE (prevents memory from being expunged on failed memory allocation)

Now let's look at the flags we can set using malloc (FreeBSD):

M_ZERO (sets memory to all 'zeros'), M_NOWAIT (returns value if memory cannot be set immediately), M_WAITOK (indicates okay to wait for memory to be set), M_USE_RESERVE (indicates memory can be set in memory reserved for OS use)

Take a good look at AllocMem and malloc. If I didn't tell you that one function was used on a non-MP OS and the other was on a MP OS, would you have been able to tell just by looking at the API call structure? No.

Similarly, the API call itself doesn't care if it's running on a MP OS or non-MP OS. There is no 'anti-MP' flag for AllocMem. Resource tracking is a separate issue from API calls, it is what the kernel handles.

Thoughts?

[HenryCase]

@bloodline
I'm glad we're getting somewhere now, you were right I was unaware of the pointer usage in AmigaOS.

bloodline wrote:
As an example,

Two task in AmigaOS... A and B

Task A will allocate some memory and fill it with some Data, and then send a pointer to Task B.  Task B receives that pointer and uses it to access the Data put there by Task A. This is a very quick process and requires only the memory allocated by Task A.

Then what we would need for MP would be a way of sticking the OS in-between all communications between tasks by manipulating the pointer data. First we need a way of flagging up when a pointer has been set by a program. I've used this guide to give me an overview of how pointers work:
http://www.codeproject.com/KB/cpp/pointers.aspx
So in C you set the pointer variable with a command like this:
int* pNumberOne; (for a pointer of integer type)

Now going to compare this with Pascal. Using this guide...
http://www.learn-programming.za.net/programming_pascal_learn13.html
So if you wanted to declare the same pointer in Pascal you would use these commands:
var
   pNumberOne: ^integer;

My question is this. When it comes to compiling the C and Pascal programs will they have the same machine code value for the pointer we declared?

[HenryCase]

Piru wrote:
@HenryCase

You can't in any sensible way intercept the pointer usage, so the question is irrelevant. You can't "flag" or any other way know when the app is using pointers and/or if it's using pointers correctly or not.

No sensible way intercept pointer usage? This is linked to the 'can it be done elegantly' question I mentioned previously, whereas this discussion is still at the 'can it be done' stage. Even if the MP solution we find is impractical, but works, then at least we'll know what our real options are.

So I'll ask again: is it possible to write a program that can scan the binary of another application and know when a pointer has been set?

[HenryCase]

@bloodline and @Piru
Thank you for your posts, I will now reply to them in one go.

bloodline wrote:
There is no way to know what the programs are passing to each other.

One thing at a time. First we need to know where the pointers are, then I'll say about my ideas on how we can use this information.

Piru wrote:
And I answer again: No.

Please explain the reasoning behind your answer, a simple 'No' doesn't help me understand why you don't think there is a way to see pointers in binary code.

Whilst I wait for your explanation, I would like to ask you another question. How does a CPU know that a program is asking to set up a pointer?

bloodline wrote:
No way, period. You could make assumptions but at best you would only be right, less than 30% of the time.

ASM is closely linked to the structure of machine code, correct? Also, ASM strives for efficiency. I would argue that there are only so many ways to set a pointer in ASM efficiently, and therefore only so many binary structures we would need to scan for to get a higher accuracy than 30% (where you got that figure from I don't know). I found a couple of ways of setting pointers in x86 ASM (I'm assuming it's x86 ASM anyway) from this page (Google cache version):
cs.colgate.edu/faculty/nevison/cs201web/lectureNotes/arraysandpointers.ppt
========================
clear2:
add  $t0, $a0, $zero    # p = addr of array[0]
forp:   sw   $zero, 0($t0)      # memory[p] = 0
addi $t0, $t0, 4        # p = p + 4
sll  $t1, $a1, 2        # $t1 = 4 * size
add  $t1, $t1, $a0      # $t1 = addr  array[size]
slt  $t2, $t0, $t1      # p < &array[size] ?
bne  $t2, $zero, forp   # if so, continue
endfor:
=========================
Pointer version assembly code
clear2:
add  $t0, $a0, $zero    # p = addr of array[0]
sll  $t1, $a1, 2        # $t1 = 4 * size
add  $t1, $t1, $a0      # $t1 = addr  array[size]
forp:   sw   $zero, 0($t0)      # memory[p] = 0
addi $t0, $t0, 4        # p = p + 4
slt  $t2, $t0, $t1      # p < &array[size] ?
bne  $t2, $zero, forp   # if so, continue
endfor:
========================
Why is it impossible to scan for the binary versions of the ASM code above?