How to move AROS forward

[bloodline]

SamuraiCrow wrote:
@HenryCase

To scan for those binary equivalents you'd have to know the difference between code and data.  If you write a patcher to go through and change all of the binary instances of memory access outside the main chunk of code, you'd probably end up inadvertently changing some constant data to something different.

I would actually be surprised if you even could identify the memory access, you have no way to know what and where that pointer is going... thus why a JIT can only translate code blocks up to the jumps.

In short you'd have to go into every program by hand and write patch code for every instance of pointer access.

And even if you could do that, it doesn't solve the problem... AmigaOS programs are designed at a fundamental level to interact intimately, you would have to rewrite the program completely.

Why not just get the source codes of everything you'd ever want to run on AROS and recompile it for MacOSX?  You'd save yourself a lot of headaches and the rest of us a lot of hassle. :-x

I agree... but you'd have a hell of a time porting a program from AmigaOS to MacOSX... I would rather just rewrite the program.

[bloodline]

XDelusion wrote:
I could go for a port of PrBOOM and zDOOM (if not GzDOOM) while we are waiting on updates.

AROS has both Doom and Quake ports, that work really well... :-)

[bloodline]

HenryCase wrote:

Bingo. CPU emulation is part of my plan. Yes it is too slow to use for running programs in real time, but it can be used to construct a 'map' of the program. This 'map' would only need to be created once, but would then be used by the operating system to help guide the program execution at key points.

If you could build a map, you wouldn't need Memory Protection. If you can build a map then you already know every possible outcome of running the program. If fact you wouldn't ever need to run the program, instead you would just look at the end of the map and see the outcome of the execution! And when you see it like that you realise that the map would be gigabytes in size for even the simples program.

[bloodline]

HenryCase wrote:
@Piru
As I said before, I am currently looking of ways around this issue.

First you need to understand the problem... at the moment you are looking at the symptoms of the problem and trying to make the symptoms go away, without actually sorting out the underlying problem.

The code map may end up being in a more high level modular structure.

A map is a map... I don't think you understand the scale of what you are suggesting!

Maybe also we keep a record of the code paths we have taken before, and if a new one arises we track its performance, adding to our code map.

Imagine you are trying to plot the Code path for a game, lets take Quake... you would have to make a code path for every single possible variation in movement made by the player, at every frame of the game... I don't think that you could map that in a thousand years...

How do you deal with prorgams that are entirely intimate... one map for both programs? one map for each program, but which have common parts? How do you deal with the fact that in AmigaOS, everything essentially runs as one big program?

These ideas are only rough sketches for now, I only realised through this thread that I'd have to deal with pointer issues when retrofitting Amiga operating systems and apps with MP, so forgive me for not going into more detail.

A Sketch it may be... but already you should be able to see that it's a flawed idea, and one that actually does nothing to solve our initial problem... which is that the OS has no way to know what a program is up to and hence no way to enforce any restrictions upon those programs.

@all
Whilst I carry on sketching out how to track the performance of Amiga programs, I'd like to take the discussion about MP on a simpler route. Seeing as the whole idea of AROS being OS 3.1 compatible is on a source code level (apart from the plans for binary compatibility on the 68k port), let's start discussing how to graft full MP into AROS through program compilers.

This doesn't make any sense. A compiler cannot add memory protection to a program... Just as a dictionary does not make for good bed time reading.

Would there be any way to make AllocMem (and all other AROS functions) behave as they should without using pointers (maybe using some new 'port' system instead that AROS could flag and control)?

Do you even know what a pointer is? How can AllocMem() work without returing a pointer to the memory you have just allocated? Could you write a story without using verbs?

Have you googled MicroSoft's Singularity yet? I think there is an OS much better suited to your needs than Amiga OS ever will be...

[bloodline]

But henry, you are missing the point. The OS already knows when a message is sent. The OS has quite a bit of information regarding message actually! But the OS does not know what the contents of the message is or how it is to be used... The compiler doesn't know this either.

The problem comes when that message contains references to memory areas not in the context of the receiving task... In AmigaOS this is fine... In an MP OS this is bad.

And right now we are only talking about Data... What if one task jumps to some code that is part of another task's context? BANG... One dead task... And exec is built around this concept!

[bloodline]

HenryCase wrote:
@bloodline and @Hans_
Thank you both again for informative posts. I think you misunderstand my plan, but that's understandable as I haven't explained it fully.

I understand your plan perfectly!

Think of it like API translation at the compiler level. So instead of declaring AllocMem() as a pointer, the same attributes about the memory requirements (memory size, flags) get declared in a malloc kind of way. What I aim to do is take the API calls and turn them into ones that can support MP. Does this sound more reasonable?

What you haven't realised is that the function call (in this case AllocMem() ) isn't the problem. It is just an interface, the problem is how one is allowed use the API. That was defined by the behaviour of the OS and apps, in such a way as to prevent the ability to retrofit Memory Protection.


In an MP OS, one would define that the pointer returned by AllocMem() is for the use of the calling task only.
In AmigaOS the pointer returned by AllocMem() is free for any task to use, this is defined behaviour.

If you change the above definition you no longer have an AmigaOS compatible system, and everything would have to be rewritten, all the various subsystems would need to be altered in the way they work to the point that no programer would recognise the system as AmigaOS... It would probably look much more like Linux. And no Amiga Programs would compile for it anymore either...

The next problem we have is that function calls in AmigaOS are simple jumps from one executable to another... in an MP OS you have to trigger an interupt, so the machine can enter supervior mode, and then deal with the data as required.

@koaftder
Cool. I'll check out your VM when I get time ("99 bottles of beer on the wall, 99 bottles of beer...").

First I tihnk you should get to grips with how AmigaOS works... get a pen and paper and plot the interactions, once you see it written down the problems will be plain to see!

[bloodline]

Hans_ wrote:

bloodline wrote:

In an MP OS, one would define that the pointer returned by AllocMem() is for the use of the calling task only.
In AmigaOS the pointer returned by AllocMem() is free for any task to use, this is defined behaviour.

If you change the above definition you no longer have an AmigaOS compatible system, and everything would have to be rewritten, all the various subsystems would need to be altered in the way they work to the point that no programer would recognise the system as AmigaOS... It would probably look much more like Linux. And no Amiga Programs would compile for it anymore either...

It's not quite that bad. Amiga OS 4 programs can request private memory with the down-side that it's per task. There is no concept of multiple threads inside a process (or sub-processes, or whatever), so you can't perform direct sharing within a multi-threaded program. In a full MP OS, memory would be private by default, and you'd have to explicitly ask for shared memory. Henry is suggesting getting the compiler to insert the MEMF_SHARED flag (and/or whatever else woulld be used) whenever it's necessary. The problem is that finding out when it's necessary is incredibly complex.

There is no reason to completely drop message passing by reference. The only requirement would be that the message doesn't contain pointers to private memory.

Hans, please don't complicate things! I'm well aware of more advanced strategies used to improve performance, but Henry is having a hard enough time grasping the basics here. He needs to understand that the messages and message passing are not the problem, it's the contents that's the problem.

So I would prefer it if we could keep a examples as simple as possible.

The next problem we have is that function calls in AmigaOS are simple jumps from one executable to another... in an MP OS you have to trigger an interupt, so the machine can enter supervior mode, and then deal with the data as required.

Ok, now you've lost me. On all OSes, function calls to libraries are simple jumps.

I'm not talking about libraries, I'm talking about how you make a system cal to the Kernel. On AmigaOS Kernel calls are simple jumps the same as a library, but on a Real OS, that has a separate kernel space, you then need to use an interrupt to jump into Kernel space...

Sending messages between processes require an interrupt. However, that's exactly the same in Amiga OS. Amiga OS programs don't just suddenly start executing each other's code within their own task/process. If a message is sent, one task/process is put to sleep (at the interrupt), and the other is woken up by the OS so that it can respond.

But calling the message function does not trigger a reschedule, AFAIR... thus does not interrupt...

Anyway, my point was to show how separate address spaces actually work... Henry is missing a fundamental point of MP. It is this point we need to address.

[bloodline]

Hans_ wrote:

bloodline wrote:

Ok, now you've lost me. On all OSes, function calls to libraries are simple jumps.

I'm not talking about libraries, I'm talking about how you make a system cal to the Kernel. On AmigaOS Kernel calls are simple jumps the same as a library, but on a Real OS, that has a separate kernel space, you then need to use an interrupt to jump into Kernel space...

Ok.

Sending messages between processes require an interrupt. However, that's exactly the same in Amiga OS. Amiga OS programs don't just suddenly start executing each other's code within their own task/process. If a message is sent, one task/process is put to sleep (at the interrupt), and the other is woken up by the OS so that it can respond.

But calling the message function does not trigger a reschedule, AFAIR... thus does not interrupt...

The other task won't receive the message until a context switch (requiring an interrupt) occurs.

Of course :-) But the actual message passing does not trigger one.. I could be wrong, here it's not really very important, we are just arguing semantics, so hopefully Henry will ignore this part of the thread.. I'll check the RKMs later and confirm.

Anyway, my point was to show how separate address spaces actually work... Henry is missing a fundamental point of MP. It is this point we need to address.

There's no need for separate address-spaces. Full memory-protection can be achieved without it.

Lets not confuse Henry anymore, please :-) We need to present the simplest example, since that gives the clearest example of why AmigaOS can't have MP...

[bloodline]

HenryCase wrote:

Hans_ wrote:
In theory it's all possible simply because we know that humans can do it.

I think that's one of the factors that is driving me on.

bloodline wrote:
In an MP OS, one would define that the pointer returned by AllocMem() is for the use of the calling task only.
In AmigaOS the pointer returned by AllocMem() is free for any task to use, this is defined behaviour.

Okay, so Amiga software expects to be able to use pointers defined by other Amiga software.

Right. Data structures (which the pointers point to) can be shared between tasks and system libraries.  In an MP OS you can't share the data structures.

So there is almost a pool of memory pointers.

No, memory pointers are nothing but the address of the data structure in memory. For example on a real Amiga I know that $DFF180 always points to colour register 0, so I can write to that address if I want to alter that colour register... you see a pointer is nothing but a number, a programmer could allocate some memory, and then write anywhere he chooses within that area, a pointer to which would be an arbitrary number. If a program needed to share that pointer it could use that arbitrary number in a function call or pass it to another program... the OS has no way to know that that arbitrary number is a pointer... does that make sense?

Can we put these memory pointers in an OS controlled environment and control which programs we'd like to use these pointers?

No, there is nothing special about these pointers.

The nearest thing to what you are suggesting is a sandbox...

In the place of the pointers in the application code we put a 'port' to the shared pointer area.

No, the CPU doesn't use 'ports'... it just pushes numbers around, a pointer is just a number that points to a data structure.

When a program then wants to access the data or instruction in memory a request to the OS goes to through that port. The OS then uses the pointer itself, and the outcome of this action is stored in a 'quarantine' area in memory. The OS then tries to determine (based on past experience and/or user preference) whether that data should be given to the application.

The program doesn't make a request. The program tells the CPU to load some data. No requests are made, the CPU just does it.

But you still miss the point. If you know you have a pointer then find there are ways and means to trap it, etc, its very slow... but the take home point here is that there is NO WAY TO KNOW WHAT IS A POINTER and WHAT IS JUST DATA :-)

Way I see it, if a program causes a computer to crash, we should be able to determine when it crashed, and what was the last value (data or instruction) to be passed for processing, right? If we can do this, we have options when it comes to crash protection (which is the main benefit of MP IMO), as long as we catch the value before it is passed for execution (or maybe before storage in its designated memory location).

When an Amiga program goes wrong, it can randomly write anywhere in the memory. This puts the computer in an unknown state, once the machine is in an unknown state you can't recover, unless you have a full backup of the Memory.

Memory protection means, that when the program goes wrong, it can only destroy it's own data/code... Only that task is in an unknown state.

Or maybe we don't have those options... can you give me typical low level examples of what causes a computer to crash?

If the programmer has made a mistake, then the CPU might write some data over some important operating system code.

Hans_ wrote:
What exactly is wrong with allowing old software to be fully-allocated in shared-memory and run in a sand-box?

I want to see the AmigaOS 3.1 API succeed as much as possible as I see it as a bridge between different Amiga-based OSs. A sandbox is okay, but it would split the community again as there would be a new 'native' API. AROS is in the unique position to bring the Amiga world together, and I don't want to see it lose that.

The AmigaOS 3.1 API is common to all AmigaOS clones.. so that's that...

Hans_ wrote:
Henry is suggesting getting the compiler to insert the MEMF_SHARED flag (and/or whatever else woulld be used) whenever it's necessary. The problem is that finding out when it's necessary is incredibly complex.

Correct, I'm advocating making memory private by default. The 'finding out when sharing is neccessary' task I am trying to pass to humans.

But nothing would work if memory was private by default, because  MEMF_PRIVATE was never enforced on AmigaOS it was ignored... not just by app programers but the operating system designers too.

bloodline wrote:
He needs to understand that the messages and message passing are not the problem, it's the contents that's the problem.

So I would prefer it if we could keep a examples as simple as possible.

bloodline, I appreciate your approach here. I agree it is best to keep the information you are all giving me as simple as possible, I do feel like I am slowly getting my head around the larger issues here, thank you for your patience (and everyone elses). The thing is I'm finding this discussion rather interesting, and as I'm getting more information it's throwing up more questions and possibilities in my head. Please forgive/understand my curiosity!

My big concern is that you are attempting to solve a problem, without having an understanding of basic operating system design or even a simple grounding in assembler.

I think though you are missing the point, because you don't understand how a computer actually works. I think you really need to learn assembler, 68k is very easy and really fun... there are loads of good books both online and in good computer stores.

[bloodline]

warpdesign wrote:

Amiga has its limits and we have to live with it.

Why should we have to live with it ?

Because this is as far as the AmigaOS API can go.

Especially now that we have a true memory protected kernel with Quark ?

:-? Quark isn't AmigaOS, it's modern microkenel. It's also proprietory, if we wanted to move the Amiga platform forward I would prefer an opensource solution. L4 is a cool system, but if you really want a MicroKernel, then I would probably choose something like Darwin, which has more real world useage and is being activly developed by comercial interests.

Wouldn't it be nice to have Aros hosted under L4 and extend from there...

What is the problem with using the Linux Kernel? It's far better supported than L4, I think it was Staf who proposed a minimal Linux with AROS-Hosted on top. You've never know Linux was there and you get all of Linux's Driver support... nice idea really.

[bloodline]

lou_dias wrote:
Back on topic...

How to improve AROS:
Instead of the wants and desired being separate, come together, form a consortium, pool the bounty money and decide in order which bounties AROS NEEDS and prioritize them.  Then let programmers bid on only the top one.  They declare how much money it would take them to code it...  Then onto the next bounty if their is money left over...  Etc...  So 3 coders offer to code the top bounty, coder 1 says he'll do it for $3,000, #2 for @$2,500 and #3 for $2,750.  Bids can be private, bounty is publicly assigned to #2...

A bounty doesn't work like that. If I put $10 into a bounty it's for a spcific task, for something that I want.

Perhaps we could set up a GeneralBounty, that people can submit to, and let a "board of directors" decide where the Money goes depending upon what is considered most important. But honestly, I doubt it will be as sucessful as what we have now. There is no money in the Amiga platform anymore, what we do now is totally for fun... that includes funding bountys and developing for the various AmigaOS Clones.

[bloodline]

cicero790 wrote:
Bloodline wrote about using the Linux kernel.
Defender of the faith about using the L4.

To Defender of the faith: What is your vision for the advantages of using L4?

From a geeky point of view, I'd prefer L4 as well, it has some great and unusual concepts... If we were setting up a company, that had investment and was going to produce our own hardware (i.e. of known configuration, and specific hardware) then I'd be keen to use L4.

But Linux has the advantage of software/hardware support and has been proven in the real world, plus AROS is already running quite happily on it, and integrated with it. :-)

Of course the advantage of this approach is that you can even swap out the underlying kernel, put something else there and all the AROS software would still run fine... or even lose the kernel altogether and let AROS run on it's own :-)

[bloodline]

hardlink wrote:
Going back to the original post:

uncharted wrote:
Start off as simply a bunch of replacement files that goes on top of an OS3.1 install and keep building them up until you have replaced everything.  ...  Really, it's how AROS should have been done from the start.

I don't follow AROS that closely, but you mean this is not being done for the non-hardware banging code?! GCC should be able to produce both 68K and x86 files; load the 68K on a hardware Amiga in place of the regular library or whatever, test the heck out of it, and if it passes try the other version in AROS.Isn't this rather standard procedure for trying to develop identical functionality on two different platforms?

This is being done... check out AFA by Bernd

[bloodline]

uncharted wrote:

bloodline wrote:

:-? Quark isn't AmigaOS, it's modern microkenel. It's also proprietory, if we wanted to move the Amiga platform forward I would prefer an opensource solution. L4 is a cool system, but if you really want a MicroKernel, then I would probably choose something like Darwin, which has more real world useage and is being activly developed by comercial interests.

Darwin is not a kernal it is the OS (and XNU is not a strict microkernel)

Sorry, I don't often get the chance to be pedantic these days, have to grab every opportunity ;-)

You are quite right, but I was trying to think of a system that would bring with it Drivers/Hardware support... and yet was still based on a Microkernel (though I realise Mach is now really a hybrid)...

What is the problem with using the Linux Kernel? It's far better supported than L4, I think it was Staf who proposed a minimal Linux with AROS-Hosted on top. You've never know Linux was there and you get all of Linux's Driver support... nice idea really.

Why not go the whole hog and host UAE with 68k AROS on Linux and get full compatibility?  That's what I've been trying to propose.

That's what I think most sensible people would want...

[bloodline]

uncharted wrote:

bloodline wrote:

This is being done... check out AFA by Bernd

This is not what AFA is doing.  AfA only replaces a couple of libs in quite an awkward way.  It is not some kind of systematic 68k port.

It's not systematic, but it's achieving the same result... the number of bugs that have been fixed by this method is quite considerable...

[bloodline]

paolone wrote:
Just a little news worth mentioning:

http://www.power2people.org/bounty_018.html

M. Schulz, after his successfull SAM440EP port, is porting AROS to the Efika platform too.

Ahh, so it won't be long before Moto has a decent OS to run on his Lego monstrosity? ;-)

Ok then, two decent OSs... :roll: :lol: