How can executables work when being thrown into different addresses on 68000 ?

[itix]

All those people who claim you can't introduce memory protection in amiga OS without breaking backwards compatibility. They make this claim because of taking memory protection by virtual address spaces as used in Windows and UNIX/Linux as a reference.

You must have misunderstood memory protection discussion on Amiga forums. You can't introduce memory protection in Amiga without breaking backwards compatibility and it has nothing to do with separate address spaces. AROS developers explained it quite we'll but can't find thread at the moment.

[itix]

This insight of AmigaOS-68k and MMU less processors now means I know how to make this happen in other environments. Decoding ELF binaries with relocation table and making use of it seems however way more complicated than the Amiga Huff format. The loader has to copy the ELF binary code into memory and then find the relocation table and interpretate it.

It is not difficult. I wrote ELF-loader for my own project long ago and ELFs are easy to parse. You might want to be careful wit relocations because subtle bug in relocation code may result in subtle bugs in software you are loading to memory.

Maybe biggest problem is finding documentation on internet. It is not a topic routinely covered in stackoverflow.com and similar sites.

[itix]

Only because software doesn't use MEMF_PUBLIC for memory that needs to be accessible by everyone (because it's never been mandatory).
 
MEMF_PUBLIC: Memory that must not be mapped, swapped,
or otherwise made non-addressable. ALL
MEMORY THAT IS REFERENCED VIA INTERRUPTS
AND/OR BY OTHER TASKS MUST BE EITHER PUBLIC
OR LOCKED INTO MEMORY! This includes both
code and data.

Commodore never documented what memory is referenced via interupts or by other tasks. API documentation has many caveats where memory should be allocated with MEMF_PUBLIC but fails to mention it. Decision was left to developers who cant have enough knowledge to understand this requirement. For example it is not enough to allocate public message port with MEMF_PUBLIC flag. Also the name in public message port must be in MEMF_PUBLIC memory to work but most developers oversee this subtle bug. Neither documentation reminer developers to copy interrupt routines to MEMF_PUBLIC memory but they should. LoadSeg() loading code to MEMF_PUBLIC memory is undocumented feature.

Other problem lies with dos.library where read/write buffers are passed to filesystem task. To work properly developer should allocate temporary MEMF_PUBLIC buffer where data is copied to/from when performing read/write. OS4 developers fixed this documentation bug by moving file operations out of filehandler tasks to task calling dos.library functions so you dont have to use MEMF_PUBLIC memory with I/O operations anymore.

Other issues arise with user interface where UI code is executed in input.device context. If developers fail to see that they might be accessing non-public memory in a context of another task.

AmigaOS could never support MEMF_PUBLIC because its API design is broken.

[itix]

Had the OS freaked out when you shared a nonpublic memory block... Adding memory protection would have been so much easier... If I recall correctly, even the commodore engineers wilfully ignored the MEMF_PUBLIC flag :-/

It was right decision. I stopped using MEMF_PUBLIC flag because it could not be enforced anymore and OS developers never documented its proper usage. It is also easier since you can allocate FileInfoBlock or temporary Exec Messsages from the stack right away.

Even if I tried to follow rules there is no guarantee my software would work properly because it was not tested in such conditions.

[itix]

I do think it is possible for new compiled programs to indicate to the OS they are memory protected (and resources tracked, another feature heavily wanted).

This is not enough because many private structures to user applications are made public by the operating system. For example struct Window to represent open window is a public structure (writable to everyone). This requires fixes to Intuition which in order can render existing software unusable.

And this is just one tiny example.

There likely will be some restrictions in usage of certain functions or some existing functions will behave different when called from a memory protected program. f.ex. auto stack extension will need that only private data is put on the stack.

In MP system there should be no concept of public memory at all. Public as in "memory that any task can alter at any time". There is no true memory protection if task Y by accidentally writing data to address X can cause entire operating system to crash. For example try memset(SysBase, 0, 0xffffffff) and see if the system can survive from that.

As a first step I see the introduction of memory contexts and make memory pools so that they can be inaccessible from certain memory contexts, read-only from others and read-write from again others. I you have done this, you have basically introduced memory protection in the OS (unfortunately no users though  ).

I would just forbid use of low level APIs like Intuition and migrate software base to use higher level abstraction (i.e. MUI). Introducing more memory types is just insane adding more unnecessary complexity. AllocMem() with MEMF_ANY should be enough.

[itix]

It doesn't require fixes to Intuition, it merely requires that you allocate the memory as public. It would be entirely possible to enforce it for new programs and not for old programs, by forcing old programs to have MEMF_PUBLIC for all their allocations.

Unfortunately your operating system stops functioning if structures are made private. So you have to fix Intuition and it would be better if the software was not using this low level component in the first place. Many of its design issues could be hidden to a high level abstraction layer.

But it is academical anyway because we know nobody is ever going to create an OS with MP compatible with existing software.

[itix]

I don't see what is wrong with providing public memory in a MP OS. Programs should be able to not need it or use it though.

The operating system should not have system critical structures in public memory. It should be possible make memory shared with other processes to at request but not by default.

SysBase could be read-only for the MP programs and they will just be killed if they try to do the above. The non-MP programs can still step on each other toes but won't be able to bring down the MP ones and the part of the OS that is also MP converted. I don't see why some clever MMU trickery shouldn't be able to do that.

Easier said than done. Libraries manipulate libbase data so there must be way to unprotect libbase temporarily. Doable but again you have to go through system code to fix it. We could accept that SetPatch() would not be permitted anymore (since it assumes public memory model) but the OS is using SetPatch() at few places so you need to redefine some other code again (mostly locale/utility stuff, IIRC).

You can always find a solution to one problem but it is likely to introduce two more. The sun explodes before all that code is fixed and in place with backward compatibility. IMO

I would not go that far and you also want to retain the possibility for devs to program their own gadgets which will need lower level functions.

You dont need Intuition for that. Its way to define gadgets is not MP friendly anyway. It is tricky when user application structures become part of system structures like struct Gadget would be. This could be fixed by copying data to system private structures but then you will find yourself rewriting bulk of Intuition again...

Or alternatively run Intuition in user space so all structures would be private. But then we could use something more sophisticated UI framework.

I don't want the GUI framework become part of the kernel.

It doesnt have to. Many BOOPSI based GUI framework could easily run in user space. There is enough abstraction to hide system internals what Intuition does not.

One other thing I also certainly want to keep is messaging by pointer passing. IMHO an OS not doing this can't be called an Amiga-like OS.

That is no problem but messages should not be public to everyone. To support this scheme messages should be allocated in a manner that they can be shared with target task (as designated by mp_SigTask) but not all message ports have designated target task.

On AROS mp_SigTask is even designated as union with callback pointer (when you PutMsg() to this kind of port OS calls this function). This PA_CALL extension is very neat feature but not actually MP friendly. I think it is perfect example when shared memory model encourages defining features which are difficult to replace without breaking compatibility both binary and source level.

[itix]

I'm not 100% sure what you mean.
 
If by that you're saying that not specifying MEMF_PUBLIC on memory that needs to be accessible by another task will cause the system to become unstable, then I agree. However we're not discussing that issue at all.

No, you did not understand it at all. The operating system need that tasks allocate some memory with MEMF_PUBLIC to be accessible by operating system tasks and handlers. There is no clean separation of user space and kernel space.

The operating system stops functioning if a program calls Disable() or Forbid(). Memory protection would prevent some accidental or malicious memory accesses from going unnoticed.

Just try something like this:

int main(void) { AddPort(malloc(sizeof(struct MinNode))); return 0; }

Next call to FindPort() may or may not hang.

You can add memory protection as much as you want but with this code operating system stops functioning.

Or maybe just:

void *gadget = AllocMem(sizeof(struct Gadget), MEMF_ANY); // Memory type is not documented in AddGadget() !
InitGadget(gadget); // init gadget to sane values so it wont crash imemdiately
AddGadget(winptr, gadget, 0);
FreeMem(gadget, sizeof(struct Gadget));
Wait(0);

You will find out that input.device is reading from unallocated memory and your keyboard and mouse is dead. Our software, however, is still runnable. It is also possible input.device is now trashing some innocent memory.

At any time we can inject toxic structures to the system. We dont have to do it deliberately: it could be just an error in the program logic.

Arguing for a robust operating system where one task can't take down the whole machine is a whole different ball game & trying to solve those issues would involve throwing everything away and starting again.
 

Isnt this what memory protection should do? Anything less is just fake if we dont count drivers and such.

[itix]

4) Is virtual memory possible? Not for Os structures. Only for private memory. Reasons are basically that swapping in memory requires a task switch, thus breaks any Forbid() and Disable(), and the Os is not prepared for that. However, if a program uses private memory (via the memory.library) such memory *can* be swapped out, and loaded from disk (automatically), again - of course - breaking the Forbid() state. Just that there are no programs using the mechanism.  Thus, a lot of plans and a lot of mechanisms have been created, but since the software basis and the user basis broke away, all the mechanisms and constructions remained unused.

Long. long ago (end of 90s I believe) I was thinking of using memory.library to allocate private memory that could be swapped away. But I realized that due to access restrictions it is not safe to pass private memory to any system call. Another problem that if I was using swappable memory it could work fine on my machine even if I violated access restrictions. On some another machine low on memory it could crash because memory was swapped away. And last and maybe also least reason: it would give nothing to WinUAE users...

So I scrapped that idea.

[itix]

This must mean that programs that has the PURE flag is feed with a parameter that tells it where this run of the code has it's variables etc.

Pure flag only indicates this program can be made resident (preloaded to ram with resident command). Internally programs can have different means to achieve re-entrancy. It could be semaphore or dynamically allocate ram for each instance.

SAS/C and GCC which have built-in support to build re-entrant (pure) code. They use custom startup code to copy data segment to newly allocated space. Pointer to data segment is placed to a4 and code is compiled to address variables relative to a4. Some special care is needed by a programmer, though.

You can of course choose to develop your program to be re-entrant without using any compiler support but then you can't use many c library calls.