How can executables work when being thrown into different addresses on 68000 ?

[itix]

All those people who claim you can't introduce memory protection in amiga OS without breaking backwards compatibility. They make this claim because of taking memory protection by virtual address spaces as used in Windows and UNIX/Linux as a reference.

You must have misunderstood memory protection discussion on Amiga forums. You can't introduce memory protection in Amiga without breaking backwards compatibility and it has nothing to do with separate address spaces. AROS developers explained it quite we'll but can't find thread at the moment.

[freqmax]

The reason for asking is that a lot of devices use ARM and MIPS processors that lack any MMU. And thus won't run any fancy unix which is entirely built on MMU that provide a separate address space for each process regardless of protection. This usally means only µCLinux remains as a free OS choice.

The main obstacle to get some Unix style environment is to be able to load arbitrary code into arbitrary locations. So I remembered that the 68000 lacked MMU but still provided arbitrary program loads and multitasking. But never investigated in depth how this was done.

This insight of AmigaOS-68k and MMU less processors now means I know how to make this happen in other environments. Decoding ELF binaries with relocation table and making use of it seems however way more complicated than the Amiga Huff format. The loader has to copy the ELF binary code into memory and then find the relocation table and interpretate it. This could form an self hosting ARM environment that works similar to AmigaOS 68k minus the custom graphics and sound.

[bloodline]

You must have misunderstood memory protection discussion on Amiga forums. You can't introduce memory protection in Amiga without breaking backwards compatibility and it has nothing to do with separate address spaces. AROS developers explained it quite we'll but can't find thread at the moment.

Fats is actually Staf... Yes he of AROS

[bloodline]

The reason for asking is that a lot of devices use ARM and MIPS processors that lack any MMU. And thus won't run any fancy unix which is entirely built on MMU that provide a separate address space for each process regardless of protection. This usally means only µCLinux remains as a free OS choice.

The main obstacle to get some Unix style environment is to be able to load arbitrary code into arbitrary locations. So I remembered that the 68000 lacked MMU but still provided arbitrary program loads and multitasking. But never investigated in depth how this was done.

The multiple address space requirement for UNIX was a design choice, while it incurs a hardware complexity penalty, the advantages have been listed in this thread. uLinux shows that UNIX doesn't actually need a per process address space to work, but you lose a few features.

This insight of AmigaOS-68k and MMU less processors now means I know how to make this happen in other environments. Decoding ELF binaries with relocation table and making use of it seems however way more complicated than the Amiga Huff format. The loader has to copy the ELF binary code into memory and then find the relocation table and interpretate it. This could form an self hosting ARM environment that works similar to AmigaOS 68k minus the custom graphics and sound.

ELF is a superior format to HUNK in my opinion, it provides all that HUNK did and more. Your understanding of how an executable file format works is incomplete there is plenty to learn and you seem keen, so keep reading!

[itix]

This insight of AmigaOS-68k and MMU less processors now means I know how to make this happen in other environments. Decoding ELF binaries with relocation table and making use of it seems however way more complicated than the Amiga Huff format. The loader has to copy the ELF binary code into memory and then find the relocation table and interpretate it.

It is not difficult. I wrote ELF-loader for my own project long ago and ELFs are easy to parse. You might want to be careful wit relocations because subtle bug in relocation code may result in subtle bugs in software you are loading to memory.

Maybe biggest problem is finding documentation on internet. It is not a topic routinely covered in stackoverflow.com and similar sites.

[psxphill]

You can't introduce memory protection in Amiga without breaking backwards compatibility and it has nothing to do with separate address spaces.

Only because software doesn't use MEMF_PUBLIC for memory that needs to be accessible by everyone (because it's never been mandatory).
 
 
MEMF_PUBLIC: Memory that must not be mapped, swapped,
or otherwise made non-addressable. ALL
MEMORY THAT IS REFERENCED VIA INTERRUPTS
AND/OR BY OTHER TASKS MUST BE EITHER PUBLIC
OR LOCKED INTO MEMORY! This includes both
code and data.
 
You could easily make it mandatory for data that needs to be passed between processes to have MEMF_PUBLIC and protect everything else. I believe there was software on the Amiga that did that and supported virtual memory as well. It wasn't that popular because most people didn't need the overhead of virtual memory and a lot of software wouldn't run with it. I think it would be quite easy to patch legacy applications to provide the correct flags though. You could also add a compatibility mode that forces all memory allocated by a task to be allocated with MEMF_PUBLIC.
 
It doesn't protect much, because a lot of data needs to be accessed by multiple tasks and there is currently no call to allow you to allocate memory that can be accessed by your task and one/many specific tasks.
 
Once you have MEMF_PUBLIC then you might be able to do per task address spaces as well.
 
The other problem you have is that because of no resource tracking, if an application does crash due to an invalid memory access it's still going to take the machine down anyway. Which might be worse than silently letting it corrupt memory & is certainly a much harder problem to fix.

[bloodline]

Only because software doesn't use MEMF_PUBLIC for memory that needs to be accessible by everyone (because it's never been mandatory).
 
 
           MEMF_PUBLIC:    Memory that must not be mapped, swapped,
                            or otherwise made non-addressable. ALL
                            MEMORY THAT IS REFERENCED VIA INTERRUPTS
                            AND/OR BY OTHER TASKS MUST BE EITHER PUBLIC
                            OR LOCKED INTO MEMORY! This includes both
                            code and data.

You could easily make it mandatory for data that needs to be passed between processes to have MEMF_PUBLIC and protect everything else. I believe there was software on the Amiga that did that and supported virtual memory as well. It wasn't that popular because a lot of software wouldn't run with it and most people didn't need the overhead of virtual memory.

One of the biggest missed opportunities of AmigaOS was the lack of Public memory enforcement...

Amiga coders, myself included, were often self taught and young... We wrote stuff that worked, rather than stuff that followed the official spec. I had no idea what the memory flag MEMF_PUBLIC was for so I ignored it and programs still worked.

Had the OS freaked out when you shared a nonpublic memory block... Adding memory protection would have been so much easier... If I recall correctly, even the commodore engineers wilfully ignored the MEMF_PUBLIC flag :-/

[itix]

Only because software doesn't use MEMF_PUBLIC for memory that needs to be accessible by everyone (because it's never been mandatory).
 
MEMF_PUBLIC: Memory that must not be mapped, swapped,
or otherwise made non-addressable. ALL
MEMORY THAT IS REFERENCED VIA INTERRUPTS
AND/OR BY OTHER TASKS MUST BE EITHER PUBLIC
OR LOCKED INTO MEMORY! This includes both
code and data.

Commodore never documented what memory is referenced via interupts or by other tasks. API documentation has many caveats where memory should be allocated with MEMF_PUBLIC but fails to mention it. Decision was left to developers who cant have enough knowledge to understand this requirement. For example it is not enough to allocate public message port with MEMF_PUBLIC flag. Also the name in public message port must be in MEMF_PUBLIC memory to work but most developers oversee this subtle bug. Neither documentation reminer developers to copy interrupt routines to MEMF_PUBLIC memory but they should. LoadSeg() loading code to MEMF_PUBLIC memory is undocumented feature.

Other problem lies with dos.library where read/write buffers are passed to filesystem task. To work properly developer should allocate temporary MEMF_PUBLIC buffer where data is copied to/from when performing read/write. OS4 developers fixed this documentation bug by moving file operations out of filehandler tasks to task calling dos.library functions so you dont have to use MEMF_PUBLIC memory with I/O operations anymore.

Other issues arise with user interface where UI code is executed in input.device context. If developers fail to see that they might be accessing non-public memory in a context of another task.

AmigaOS could never support MEMF_PUBLIC because its API design is broken.

[itix]

Had the OS freaked out when you shared a nonpublic memory block... Adding memory protection would have been so much easier... If I recall correctly, even the commodore engineers wilfully ignored the MEMF_PUBLIC flag :-/

It was right decision. I stopped using MEMF_PUBLIC flag because it could not be enforced anymore and OS developers never documented its proper usage. It is also easier since you can allocate FileInfoBlock or temporary Exec Messsages from the stack right away.

Even if I tried to follow rules there is no guarantee my software would work properly because it was not tested in such conditions.

[rewlako]

There is no such think like an EXEC loader.

Loading modules is a function of  DOS.

You know perfectly well what I mean.

[Fats]

You must have misunderstood memory protection discussion on Amiga forums.

I don't think so
If you mean that you can't force memory protection on current compiled programs I agree.
I even agree that for programs who want MP, it probably means some more work then just a recompile although changes needed I foresee seem to be much less than what most other people think.
What I don't agree with is that MP can't be introduced in an Amiga like OS without breaking backwards compatibility for already compiled programs.

I do think it is possible for new compiled programs to indicate to the OS they are memory protected (and resources tracked, another feature heavily wanted). There likely will be some restrictions in usage of certain functions or some existing functions will behave different when called from a memory protected program. f.ex. auto stack extension will need that only private data is put on the stack.
With this approach I think it should be possible to come to a state that memory protected programs will catch bug and gracefully exit without keeping resources in the system.

As a first step I see the introduction of memory contexts and make memory pools so that they can be inaccessible from certain memory contexts, read-only from others and read-write from again others. I you have done this, you have basically introduced memory protection in the OS (unfortunately no users though ).
Next step is to gradually see case by case, library by library, driver by driver how this feature can be used and with the least behavioural impact on existing code.
Now that would bring back Research in AROS and IMHO much better than finding excuses using the hardest corner cases to not start the work.

I even agree that having a hacker proof MP system e.g. that is resistant to programmers who want to deliberately circumvent the MP will be another job of much bigger proportions and likely difficult to introduce in a legacy OS like Amiga OS. But I would already be satisfied with MP as discussed in previous paragraph.

greets,
Staf.

• The 'why-don't-you-do-it-yourself' reply is unnecessary; my mind is currently occupied with other things not or only indirectly (NG) Amiga related. But if somebody feels the itch don't hesitate to contact me.

[Karlos]

Fats is actually Staf... Yes he of AROS

Some endian conversion issues in the username, huh?

[freqmax]

One could track resources and watch when the program exits to free those regardless if the program free'd them properly. And have a database of permissions for old software so one can automaticly apply restrictions.

I'm sure AmigaOS can be made more secure. But it will be a slight kludge at least for older software.

[itix]

I do think it is possible for new compiled programs to indicate to the OS they are memory protected (and resources tracked, another feature heavily wanted).

This is not enough because many private structures to user applications are made public by the operating system. For example struct Window to represent open window is a public structure (writable to everyone). This requires fixes to Intuition which in order can render existing software unusable.

And this is just one tiny example.

There likely will be some restrictions in usage of certain functions or some existing functions will behave different when called from a memory protected program. f.ex. auto stack extension will need that only private data is put on the stack.

In MP system there should be no concept of public memory at all. Public as in "memory that any task can alter at any time". There is no true memory protection if task Y by accidentally writing data to address X can cause entire operating system to crash. For example try memset(SysBase, 0, 0xffffffff) and see if the system can survive from that.

As a first step I see the introduction of memory contexts and make memory pools so that they can be inaccessible from certain memory contexts, read-only from others and read-write from again others. I you have done this, you have basically introduced memory protection in the OS (unfortunately no users though  ).

I would just forbid use of low level APIs like Intuition and migrate software base to use higher level abstraction (i.e. MUI). Introducing more memory types is just insane adding more unnecessary complexity. AllocMem() with MEMF_ANY should be enough.

[psxphill]

This is not enough because many private structures to user applications are made public by the operating system. For example struct Window to represent open window is a public structure (writable to everyone). This requires fixes to Intuition which in order can render existing software unusable.

It doesn't require fixes to Intuition, it merely requires that you allocate the memory as public. It would be entirely possible to enforce it for new programs and not for old programs, by forcing old programs to have MEMF_PUBLIC for all their allocations.
 
You could argue that this wouldn't protect much, but not that it can't be done.

[itix]

It doesn't require fixes to Intuition, it merely requires that you allocate the memory as public. It would be entirely possible to enforce it for new programs and not for old programs, by forcing old programs to have MEMF_PUBLIC for all their allocations.

Unfortunately your operating system stops functioning if structures are made private. So you have to fix Intuition and it would be better if the software was not using this low level component in the first place. Many of its design issues could be hidden to a high level abstraction layer.

But it is academical anyway because we know nobody is ever going to create an OS with MP compatible with existing software.