Amiga.org
Amiga computer related discussion => Amiga Software Issues and Discussion => Topic started by: freqmax on May 28, 2014, 05:43:30 AM
-
The M68000 CPU has no MMU and thus can't move memory around and always present a neat linear memory such that all executables start at address zero. Instead executables must by necessity be copied into an arbitrary memory position and be made to work from that memory position. However this would require that all data manipulations using an addresses within the program, jumps, etc. Must all be changed for every loaded program. So how has this been solved in AmigaOS ?
-
The executable file format, IIRC, provides a list of addresses that need to be adjusted for relocation. Also, the 68000 has a sizable range for relative branching (+/- 32KB) and highly orthogonal addressing modes, so it's entirely possible to construct code that doesn't use fixed addresses at all.
-
Alright so every executable on Amiga 0x000003F3 contains both code to be executed and address position list?
Such that when Exec is instructed to load an executable. It will copy the code to the allocated memory. Then Exec will use the table list of addresses and modify all address references..? before executing the code.
-
The Amiga HUNK file format allows you to have both relative and absolute addressing on the data chunks. A file can contain a combination of many different HUNK types.
In short, when the EXEC loader hits a relocateable chunk (for example HUNK_RELOC32, $000003f7), the contents are loaded into an allocated memory space. The loader then proceeds to work on a lookup-table (also provided by the HUNK file) which contains the addresses to adjust accordingly.
-
There is no such think like an EXEC loader.
Loading modules is a function of DOS.
-
Such that when Exec is instructed to load an executable. It will copy the code to the allocated memory. Then Exec will use the table list of addresses and modify all address references..? before executing the code.
BTW, also AmigaOS4, MorphOS and AROS use relocatable binaries.
-
By tradition or need? PowerPC has MMU asfair?
-
Yes, PPC has an MMU - but everything in the Amiga system architecture is designed around a flat global memory map. Message-passing is as simple as handing over a pointer to the message to the receiving process. So introducing hardware relocation or other MMU concepts would either require an insane amount of bookkeeping on the part of the OS, or throw everything into chaos.
-
It should be possible to use the "page table" that the MMU uses to recalculate these pointers faster than the task switch takes? But I understand the benefits of pointer messaging (brilliant!).
There is one clear benefit as I can se. By avoiding the MMU one can also remove the latency between the CPU to memory and I/O.
I had a deeper look into the Amiga executable format (http://amiga.rules.no/abfs/abfs.pdf). And if I understand this correctly. An arbitrary memory location is allocated and the instruction sequence (code) is copied there. In the next step the operating system (DOS?) will use the relocation table if any to alter any absolute code location on the fly.
Seems softkick (http://www.amiga.org/forums/showthread.php?t=64142) works in a similar manner with relocation tables. However the creation of those seems to be more magic that ordinary compiled files. Though trapping a debugger seems reasonable to find all absolute references in a ROM file.
-
By tradition or need? PowerPC has MMU asfair?
The need for MMU for multi-tasking OSes has grown when OSes evolved from single task to multi-task. Originally computers were single task with code hard linked to a certain start address. For the first multi-tasking OSes the whole process and it's state were swapped from memory and the next process image loaded. Next step was introduction of MMUs so programs could still think they were single task but actually were multi-tasking.
If you don't need this legacy there is no reason to not relocate the binaries during load. Some people think you need multiple adress spaces in order to provide memory protection. This is false, see SASOS (http://en.wikipedia.org/wiki/Single_address_space_operating_system) for more information.
-
The need for MMU for multi-tasking OSes has grown when OSes evolved from single task to multi-task. Originally computers were single task with code hard linked to a certain start address.
Relocation has been around forever. It's a must if you have code that is loadable at runtime.
Even if you have a machine that in most use loads a image to a fixed address and jumps there it's very unlikely that no one has never found a use for position independent code on it.
Next step was introduction of MMUs so programs could still think they were single task but actually were multi-tasking.
You don't need an MMU to do multitasking. MMUs are for memory protection and virtualising address spaces.
If you don't need this legacy there is no reason to not relocate the binaries during load.
If you want to have code that is loadable at runtime without switching out the complete task every time the active task changes you have to have relocation or some sort of address space virtualisation. It's far more sensible to have a table of addresses that have to be relocated by whatever loads the code than it is to move entire tasks around in memory.
Some people think you need multiple adress spaces in order to provide memory protection.
Who thinks that? Lots of microcontrollers provide limited memory protection without an MMU. I.e. write to address range X in user state causes an exception.
-
Argh!
Position independent coding has been around since the 8 bit cpu.
No MMU is needed.
Many OS' require it.
-
Some people think you need multiple adress spaces in order to provide memory protection.
Who thinks that? Lots of microcontrollers provide limited memory protection without an MMU. I.e. write to address range X in user state causes an exception.
All those people who claim you can't introduce memory protection in amiga OS without breaking backwards compatibility. They make this claim because of taking memory protection by virtual address spaces as used in Windows and UNIX/Linux as a reference.
-
However this would require that all data manipulations using an addresses within the program, jumps, etc. Must all be changed for every loaded program. So how has this been solved in AmigaOS ?
AmigaOS changes them for every loaded program. http://en.wikipedia.org/wiki/Amiga_Hunk
Who thinks that? Lots of microcontrollers provide limited memory protection without an MMU. I.e. write to address range X in user state causes an exception.
You missed his point. Most operating systems that use an MMU have separate memory spaces because it has other benefits, like avoiding memory fragmentation. Not because you need it to protect memory between tasks. If you don't have an MMU then you can't use separate address spaces or realistically protect memory between tasks.
Your limited memory protection sounds like the TLB in the 68EC040, which isn't really good enough for what AmigaOS uses the MMU for.
http://amigadev.elowar.com/read/ADCD_2.1/AmigaMail_Vol2_guide/node0161.html
-
All those people who claim you can't introduce memory protection in amiga OS without breaking backwards compatibility. They make this claim because of taking memory protection by virtual address spaces as used in Windows and UNIX/Linux as a reference.
You must have misunderstood memory protection discussion on Amiga forums. You can't introduce memory protection in Amiga without breaking backwards compatibility and it has nothing to do with separate address spaces. AROS developers explained it quite we'll but can't find thread at the moment.
-
The reason for asking is that a lot of devices use ARM and MIPS processors that lack any MMU. And thus won't run any fancy unix which is entirely built on MMU that provide a separate address space for each process regardless of protection. This usally means only µCLinux remains as a free OS choice.
The main obstacle to get some Unix style environment is to be able to load arbitrary code into arbitrary locations. So I remembered that the 68000 lacked MMU but still provided arbitrary program loads and multitasking. But never investigated in depth how this was done.
This insight of AmigaOS-68k and MMU less processors now means I know how to make this happen in other environments. Decoding ELF binaries with relocation table and making use of it seems however way more complicated than the Amiga Huff format. The loader has to copy the ELF binary code into memory and then find the relocation table and interpretate it. This could form an self hosting ARM environment that works similar to AmigaOS 68k minus the custom graphics and sound.
-
You must have misunderstood memory protection discussion on Amiga forums. You can't introduce memory protection in Amiga without breaking backwards compatibility and it has nothing to do with separate address spaces. AROS developers explained it quite we'll but can't find thread at the moment.
Fats is actually Staf... Yes he of AROS :)
-
The reason for asking is that a lot of devices use ARM and MIPS processors that lack any MMU. And thus won't run any fancy unix which is entirely built on MMU that provide a separate address space for each process regardless of protection. This usally means only µCLinux remains as a free OS choice.
The main obstacle to get some Unix style environment is to be able to load arbitrary code into arbitrary locations. So I remembered that the 68000 lacked MMU but still provided arbitrary program loads and multitasking. But never investigated in depth how this was done.
The multiple address space requirement for UNIX was a design choice, while it incurs a hardware complexity penalty, the advantages have been listed in this thread. uLinux shows that UNIX doesn't actually need a per process address space to work, but you lose a few features.
This insight of AmigaOS-68k and MMU less processors now means I know how to make this happen in other environments. Decoding ELF binaries with relocation table and making use of it seems however way more complicated than the Amiga Huff format. The loader has to copy the ELF binary code into memory and then find the relocation table and interpretate it. This could form an self hosting ARM environment that works similar to AmigaOS 68k minus the custom graphics and sound.
ELF is a superior format to HUNK in my opinion, it provides all that HUNK did and more. Your understanding of how an executable file format works is incomplete :) there is plenty to learn and you seem keen, so keep reading!
-
This insight of AmigaOS-68k and MMU less processors now means I know how to make this happen in other environments. Decoding ELF binaries with relocation table and making use of it seems however way more complicated than the Amiga Huff format. The loader has to copy the ELF binary code into memory and then find the relocation table and interpretate it.
It is not difficult. I wrote ELF-loader for my own project long ago and ELFs are easy to parse. You might want to be careful wit relocations because subtle bug in relocation code may result in subtle bugs in software you are loading to memory.
Maybe biggest problem is finding documentation on internet. It is not a topic routinely covered in stackoverflow.com and similar sites.
-
You can't introduce memory protection in Amiga without breaking backwards compatibility and it has nothing to do with separate address spaces.
Only because software doesn't use MEMF_PUBLIC for memory that needs to be accessible by everyone (because it's never been mandatory).
MEMF_PUBLIC: Memory that must not be mapped, swapped,
or otherwise made non-addressable. ALL
MEMORY THAT IS REFERENCED VIA INTERRUPTS
AND/OR BY OTHER TASKS MUST BE EITHER PUBLIC
OR LOCKED INTO MEMORY! This includes both
code and data.
You could easily make it mandatory for data that needs to be passed between processes to have MEMF_PUBLIC and protect everything else. I believe there was software on the Amiga that did that and supported virtual memory as well. It wasn't that popular because most people didn't need the overhead of virtual memory and a lot of software wouldn't run with it. I think it would be quite easy to patch legacy applications to provide the correct flags though. You could also add a compatibility mode that forces all memory allocated by a task to be allocated with MEMF_PUBLIC.
It doesn't protect much, because a lot of data needs to be accessed by multiple tasks and there is currently no call to allow you to allocate memory that can be accessed by your task and one/many specific tasks.
Once you have MEMF_PUBLIC then you might be able to do per task address spaces as well.
The other problem you have is that because of no resource tracking, if an application does crash due to an invalid memory access it's still going to take the machine down anyway. Which might be worse than silently letting it corrupt memory & is certainly a much harder problem to fix.
-
Only because software doesn't use MEMF_PUBLIC for memory that needs to be accessible by everyone (because it's never been mandatory).
MEMF_PUBLIC: Memory that must not be mapped, swapped,
or otherwise made non-addressable. ALL
MEMORY THAT IS REFERENCED VIA INTERRUPTS
AND/OR BY OTHER TASKS MUST BE EITHER PUBLIC
OR LOCKED INTO MEMORY! This includes both
code and data.
You could easily make it mandatory for data that needs to be passed between processes to have MEMF_PUBLIC and protect everything else. I believe there was software on the Amiga that did that and supported virtual memory as well. It wasn't that popular because a lot of software wouldn't run with it and most people didn't need the overhead of virtual memory.
One of the biggest missed opportunities of AmigaOS was the lack of Public memory enforcement...
Amiga coders, myself included, were often self taught and young... We wrote stuff that worked, rather than stuff that followed the official spec. I had no idea what the memory flag MEMF_PUBLIC was for so I ignored it and programs still worked.
Had the OS freaked out when you shared a nonpublic memory block... Adding memory protection would have been so much easier... If I recall correctly, even the commodore engineers wilfully ignored the MEMF_PUBLIC flag :-/
-
Only because software doesn't use MEMF_PUBLIC for memory that needs to be accessible by everyone (because it's never been mandatory).
MEMF_PUBLIC: Memory that must not be mapped, swapped,
or otherwise made non-addressable. ALL
MEMORY THAT IS REFERENCED VIA INTERRUPTS
AND/OR BY OTHER TASKS MUST BE EITHER PUBLIC
OR LOCKED INTO MEMORY! This includes both
code and data.
Commodore never documented what memory is referenced via interupts or by other tasks. API documentation has many caveats where memory should be allocated with MEMF_PUBLIC but fails to mention it. Decision was left to developers who cant have enough knowledge to understand this requirement. For example it is not enough to allocate public message port with MEMF_PUBLIC flag. Also the name in public message port must be in MEMF_PUBLIC memory to work but most developers oversee this subtle bug. Neither documentation reminer developers to copy interrupt routines to MEMF_PUBLIC memory but they should. LoadSeg() loading code to MEMF_PUBLIC memory is undocumented feature.
Other problem lies with dos.library where read/write buffers are passed to filesystem task. To work properly developer should allocate temporary MEMF_PUBLIC buffer where data is copied to/from when performing read/write. OS4 developers fixed this documentation bug by moving file operations out of filehandler tasks to task calling dos.library functions so you dont have to use MEMF_PUBLIC memory with I/O operations anymore.
Other issues arise with user interface where UI code is executed in input.device context. If developers fail to see that they might be accessing non-public memory in a context of another task.
AmigaOS could never support MEMF_PUBLIC because its API design is broken.
-
Had the OS freaked out when you shared a nonpublic memory block... Adding memory protection would have been so much easier... If I recall correctly, even the commodore engineers wilfully ignored the MEMF_PUBLIC flag :-/
It was right decision. I stopped using MEMF_PUBLIC flag because it could not be enforced anymore and OS developers never documented its proper usage. It is also easier since you can allocate FileInfoBlock or temporary Exec Messsages from the stack right away.
Even if I tried to follow rules there is no guarantee my software would work properly because it was not tested in such conditions.
-
There is no such think like an EXEC loader.
Loading modules is a function of DOS.
You know perfectly well what I mean.
-
You must have misunderstood memory protection discussion on Amiga forums.
I don't think so ;)
If you mean that you can't force memory protection on current compiled programs I agree.
I even agree that for programs who want MP, it probably means some more work then just a recompile although changes needed I foresee seem to be much less than what most other people think.
What I don't agree with is that MP can't be introduced in an Amiga like OS without breaking backwards compatibility for already compiled programs.
I do think it is possible for new compiled programs to indicate to the OS they are memory protected (and resources tracked, another feature heavily wanted). There likely will be some restrictions in usage of certain functions or some existing functions will behave different when called from a memory protected program. f.ex. auto stack extension will need that only private data is put on the stack.
With this approach I think it should be possible to come to a state that memory protected programs will catch bug and gracefully exit without keeping resources in the system.
As a first step I see the introduction of memory contexts and make memory pools so that they can be inaccessible from certain memory contexts, read-only from others and read-write from again others. I you have done this, you have basically introduced memory protection in the OS (unfortunately no users though :) ).
Next step is to gradually see case by case, library by library, driver by driver how this feature can be used and with the least behavioural impact on existing code.
Now that would bring back Research in AROS and IMHO much better than finding excuses using the hardest corner cases to not start the work.
I even agree that having a hacker proof MP system e.g. that is resistant to programmers who want to deliberately circumvent the MP will be another job of much bigger proportions and likely difficult to introduce in a legacy OS like Amiga OS. But I would already be satisfied with MP as discussed in previous paragraph.
greets,
Staf.
- The 'why-don't-you-do-it-yourself' reply is unnecessary; my mind is currently occupied with other things not or only indirectly (NG) Amiga related. But if somebody feels the itch don't hesitate to contact me.
-
Fats is actually Staf... Yes he of AROS :)
Some endian conversion issues in the username, huh? :)
-
One could track resources and watch when the program exits to free those regardless if the program free'd them properly. And have a database of permissions for old software so one can automaticly apply restrictions.
I'm sure AmigaOS can be made more secure. But it will be a slight kludge at least for older software.
-
I do think it is possible for new compiled programs to indicate to the OS they are memory protected (and resources tracked, another feature heavily wanted).
This is not enough because many private structures to user applications are made public by the operating system. For example struct Window to represent open window is a public structure (writable to everyone). This requires fixes to Intuition which in order can render existing software unusable.
And this is just one tiny example.
There likely will be some restrictions in usage of certain functions or some existing functions will behave different when called from a memory protected program. f.ex. auto stack extension will need that only private data is put on the stack.
In MP system there should be no concept of public memory at all. Public as in "memory that any task can alter at any time". There is no true memory protection if task Y by accidentally writing data to address X can cause entire operating system to crash. For example try memset(SysBase, 0, 0xffffffff) and see if the system can survive from that.
As a first step I see the introduction of memory contexts and make memory pools so that they can be inaccessible from certain memory contexts, read-only from others and read-write from again others. I you have done this, you have basically introduced memory protection in the OS (unfortunately no users though ).
I would just forbid use of low level APIs like Intuition and migrate software base to use higher level abstraction (i.e. MUI). Introducing more memory types is just insane adding more unnecessary complexity. AllocMem() with MEMF_ANY should be enough.
-
This is not enough because many private structures to user applications are made public by the operating system. For example struct Window to represent open window is a public structure (writable to everyone). This requires fixes to Intuition which in order can render existing software unusable.
It doesn't require fixes to Intuition, it merely requires that you allocate the memory as public. It would be entirely possible to enforce it for new programs and not for old programs, by forcing old programs to have MEMF_PUBLIC for all their allocations.
You could argue that this wouldn't protect much, but not that it can't be done.
-
It doesn't require fixes to Intuition, it merely requires that you allocate the memory as public. It would be entirely possible to enforce it for new programs and not for old programs, by forcing old programs to have MEMF_PUBLIC for all their allocations.
Unfortunately your operating system stops functioning if structures are made private. So you have to fix Intuition and it would be better if the software was not using this low level component in the first place. Many of its design issues could be hidden to a high level abstraction layer.
But it is academical anyway because we know nobody is ever going to create an OS with MP compatible with existing software.
-
In MP system there should be no concept of public memory at all. Public as in "memory that any task can alter at any time".
I don't see what is wrong with providing public memory in a MP OS. Programs should be able to not need it or use it though.
There is no true memory protection if task Y by accidentally writing data to address X can cause entire operating system to crash. For example try memset(SysBase, 0, 0xffffffff) and see if the system can survive from that.
SysBase could be read-only for the MP programs and they will just be killed if they try to do the above. The non-MP programs can still step on each other toes but won't be able to bring down the MP ones and the part of the OS that is also MP converted. I don't see why some clever MMU trickery shouldn't be able to do that.
I would just forbid use of low level APIs like Intuition and migrate software base to use higher level abstraction (i.e. MUI).
I would not go that far and you also want to retain the possibility for devs to program their own gadgets which will need lower level functions. I don't want the GUI framework become part of the kernel.
One other thing I also certainly want to keep is messaging by pointer passing. IMHO an OS not doing this can't be called an Amiga-like OS.
Introducing more memory types is just insane adding more unnecessary complexity. AllocMem() with MEMF_ANY should be enough.
I personally want some more fine-grained access rights than the classic process based MP as in UNIX with an additional kernel context that basically can read/write everything. I also would like something that is much less complex than SELinux for defining the access rights.
To me an amiga-like OS has to be micro-kernel and the kernel (e.g. exec.library) should be kept as small as possible. It should just provide the facilities to two tasks to have memory which they both can access but no other tasks can access but the kernel's involvement should stop there.
-
The reason for asking is that a lot of devices use ARM and MIPS processors that lack any MMU.
** snip **
This usally means only µCLinux remains as a free OS choice.
These days in situations where you don't have an MMU you usually don't have one because you aren't going to be running a general purpose OS. You are either going barebones there or using an RTOS. Most RTOS don't have loaders for executables etc because the "programs" are compiled into the same image as the "OS".
load arbitrary code into arbitrary locations.
Surely that requirement exists for any OS that can load programs at runtime.
The code isn't arbitary though. If it's going to be running with an OS it has to abide by the requirements of that OS.
Decoding ELF binaries with relocation table and making use of it seems however way more complicated than the Amiga Huff format.
I'm not sure how far you've actually researched this stuff but uClinux doesn't use ELF either. It uses BFLT. Because of how hard it is to get a target into gcc etc and have it stay working the uClinux targets do compile to ELF but that output is then passed through a tool that generates the relocation table, adds information about the entry point, stack size, whether the code should be loaded to into ram or execute in place etc. The BFLT is a lot simpler than HUNK.. It's basically a header, relocation data and then the program image.
-
I also would like something that is much less complex than SELinux for defining the access rights.
SELinux has little to do with memory protection in the context of this thread.
Unix has simpler access control systems.. you know like the one that's built in i.e. file permissions etc.
-
AmigaOS changes them for every loaded program.
You missed his point.
I don't think I did. You seem to have thought that I was talking only about AmigaOS when I wasn't. Not sure why I would bring up microcontrollers if that was the case TBH.
Most operating systems that use an MMU have separate memory spaces because it has other benefits, like avoiding memory fragmentation.
And being able to resize tasks etc. Memory protection goes hand in hand with what a MMU is doing in most systems but you don't have to have a fully fledged MMU to protect the kernel from user processes.
If you don't have an MMU then you can't use separate address spaces or realistically protect memory between tasks.
If you don't have an MMU but have some sort of memory protection you can:
Protect the kernel and it's memory. Protect memory mapped devices like NOR flash from being corrupted by a runaway user process. Depending how how fine grained it is you can protect other processes from the active process by restricting the active process to working within the memory allotted to it. I know a lot of that doesn't apply to AmigaOS.
Your limited memory protection sounds like the TLB in the 68EC040, which isn't really good enough for what AmigaOS uses the MMU for.
My limited memory protection is the type of thing that exists in a lot of recent microcontrollers and to a very limited degree in the DragonBall SoCs if it has to be 68k related.
-
I don't see what is wrong with providing public memory in a MP OS. Programs should be able to not need it or use it though.
The operating system should not have system critical structures in public memory. It should be possible make memory shared with other processes to at request but not by default.
SysBase could be read-only for the MP programs and they will just be killed if they try to do the above. The non-MP programs can still step on each other toes but won't be able to bring down the MP ones and the part of the OS that is also MP converted. I don't see why some clever MMU trickery shouldn't be able to do that.
Easier said than done. Libraries manipulate libbase data so there must be way to unprotect libbase temporarily. Doable but again you have to go through system code to fix it. We could accept that SetPatch() would not be permitted anymore (since it assumes public memory model) but the OS is using SetPatch() at few places so you need to redefine some other code again (mostly locale/utility stuff, IIRC).
You can always find a solution to one problem but it is likely to introduce two more. The sun explodes before all that code is fixed and in place with backward compatibility. IMO :)
I would not go that far and you also want to retain the possibility for devs to program their own gadgets which will need lower level functions.
You dont need Intuition for that. Its way to define gadgets is not MP friendly anyway. It is tricky when user application structures become part of system structures like struct Gadget would be. This could be fixed by copying data to system private structures but then you will find yourself rewriting bulk of Intuition again...
Or alternatively run Intuition in user space so all structures would be private. But then we could use something more sophisticated UI framework.
I don't want the GUI framework become part of the kernel.
It doesnt have to. Many BOOPSI based GUI framework could easily run in user space. There is enough abstraction to hide system internals what Intuition does not.
One other thing I also certainly want to keep is messaging by pointer passing. IMHO an OS not doing this can't be called an Amiga-like OS.
That is no problem but messages should not be public to everyone. To support this scheme messages should be allocated in a manner that they can be shared with target task (as designated by mp_SigTask) but not all message ports have designated target task.
On AROS mp_SigTask is even designated as union with callback pointer (when you PutMsg() to this kind of port OS calls this function). This PA_CALL extension is very neat feature but not actually MP friendly. I think it is perfect example when shared memory model encourages defining features which are difficult to replace without breaking compatibility both binary and source level.
-
@donpalmera, I want to be able to run a partial general purpose OS. Such that one can login edit C-files and compile them. Mostly to be able to change functionality without setting up a cross development environment. Which is not always possible.
Code loading at arbitrary positions are done by miost OS:s. The difference is that computers with MMU can use a static memory position as opposed when you actually need the relocation table.
I found the x86-32-ELF format more complicated than Amiga-68k-HUFF. I didn't have a look at µCLinux in that aspect.
As for making use of MMU-less ARM (or perhaps MIPS). It's just noted that ELF decoding and relocation might be less straightforward than I thought. But most importantly is that I know gcc + ld may and will produce relocatable binaries. How to make it happen in code is another story. The next chapter is how to make multitasking happen. And perhaps if a sufficient handleable QFP surface mount CPU with MMU can be had.
-
@donpalmera, I want to be able to run a partial general purpose OS. Such that one can login edit C-files and compile them. Mostly to be able to change functionality without setting up a cross development environment. Which is not always possible.
Code loading at arbitrary positions are done by miost OS:s. The difference is that computers with MMU can use a static memory position as opposed when you actually need the relocation table.
I found the x86-32-ELF format more complicated than Amiga-68k-HUFF. I didn't have a look at µCLinux in that aspect.
As for making use of MMU-less ARM (or perhaps MIPS). It's just noted that ELF decoding and relocation might be less straightforward than I thought. But most importantly is that I know gcc + ld may and will produce relocatable binaries. How to make it happen in code is another story. The next chapter is how to make multitasking happen. And perhaps if a sufficient handleable QFP surface mount CPU with MMU can be had.
Compile your code with the -fpic option ;-)
-
Unfortunately your operating system stops functioning if structures are made private.
I'm not 100% sure what you mean.
If by that you're saying that not specifying MEMF_PUBLIC on memory that needs to be accessible by another task will cause the system to become unstable, then I agree. However we're not discussing that issue at all.
The operating system stops functioning if a program calls Disable() or Forbid(). Memory protection would prevent some accidental or malicious memory accesses from going unnoticed.
On 68000 you can do similar bad things to the operating system by using odd addresses.
Arguing for a robust operating system where one task can't take down the whole machine is a whole different ball game & trying to solve those issues would involve throwing everything away and starting again.
If you don't have an MMU but have some sort of memory protection you can:
Protect the kernel and it's memory. Protect memory mapped devices like NOR flash from being corrupted by a runaway user process. Depending how how fine grained it is you can protect other processes from the active process by restricting the active process to working within the memory allotted to it. I know a lot of that doesn't apply to AmigaOS.
If it's fine grained enough that you can protect the active process then it's basically an MMU. Protecting a handful of ranges of memory doesn't count as memory protection (AmigaOS already does that itself).
The difference is that computers with MMU can use a static memory position as opposed when you actually need the relocation table.
I've yet to see an operating system that uses an MMU to avoid ever having to re-locate executable files. The way windows executables works is they have an address they are linked to, but there is enough information that they can be relocated. If it can load it to the address the file wants then it saves time relocating it.
That is no problem but messages should not be public to everyone. To support this scheme messages should be allocated in a manner that they can be shared with target task (as designated by mp_SigTask) but not all message ports have designated target task.
You can never can know what task needs to be able to write to your memory. If you use Read() to fill your message then the memory needs to be writable by a task created by the filesystem. To do what you propose in every circumstance isn't simple, although it could be used in some cases.
-
Suggestion, make new OS. And write an compability layer.
-
You can always find a solution to one problem but it is likely to introduce two more. The sun explodes before all that code is fixed and in place with backward compatibility. IMO :)
I just hope I somewhere find the time, inspiration and transpiration to prove you wrong ;)
-
Suggestion, make new OS. And write an compability layer.
Linux+ AROS ?
;)
-
I'm not 100% sure what you mean.
If by that you're saying that not specifying MEMF_PUBLIC on memory that needs to be accessible by another task will cause the system to become unstable, then I agree. However we're not discussing that issue at all.
No, you did not understand it at all. The operating system need that tasks allocate some memory with MEMF_PUBLIC to be accessible by operating system tasks and handlers. There is no clean separation of user space and kernel space.
The operating system stops functioning if a program calls Disable() or Forbid(). Memory protection would prevent some accidental or malicious memory accesses from going unnoticed.
Just try something like this:
int main(void) { AddPort(malloc(sizeof(struct MinNode))); return 0; }
Next call to FindPort() may or may not hang.
You can add memory protection as much as you want but with this code operating system stops functioning.
Or maybe just:
void *gadget = AllocMem(sizeof(struct Gadget), MEMF_ANY); // Memory type is not documented in AddGadget() !
InitGadget(gadget); // init gadget to sane values so it wont crash imemdiately
AddGadget(winptr, gadget, 0);
FreeMem(gadget, sizeof(struct Gadget));
Wait(0);
You will find out that input.device is reading from unallocated memory and your keyboard and mouse is dead. Our software, however, is still runnable. It is also possible input.device is now trashing some innocent memory.
At any time we can inject toxic structures to the system. We dont have to do it deliberately: it could be just an error in the program logic.
Arguing for a robust operating system where one task can't take down the whole machine is a whole different ball game & trying to solve those issues would involve throwing everything away and starting again.
Isnt this what memory protection should do? Anything less is just fake if we dont count drivers and such.
-
SELinux has little to do with memory protection in the context of this thread.
I have to admit I am no SELinux expert (I just disable it) but what I had in mind when I wrote that is that you use shared memory if two processes want common memory access and SELinux is used to restrict access right to this shared memory.
-
Probably a couple of concepts are mixed here. Please allow me to explain:
1) Memory *protection*: If you mean by that that certain areas of memory can be prevented from getting overwritten, yes, this is perfectly possible. If you load kickstart modules with "LoadModules", you can protect them from getting overwritten by MuProtectModules, this is what the command is good for. If you write a program and want its code segment protected from getting overwritten, you can do so by post-processing with MuLink. All the features are provided by the mmu.library back then.... of course, nobody ever used MuLink, but MuProtectModules is in active use.
2) If you mean *isolation* of programs, i.e. protect the address space of program a) from messing with the address space of program b), then there are only limited means (but *some* means) to do that, but programs need to request this explicitly. Problem: The Os moves its data structures around in global memory and has no resource management. Meaning, it has no control which process "owns" a resource. Thus, every program that interacts with the Os *necessarily* has to see parts of the global address space simply because that's where the Os keeps all its data. There is neither a problem with program a) allocating a resource, passing it over to program b) which uses it, and program c) releasing it. As long as this is consistent.
3) Can programs still isolate *parts* of their data? Yes, they can. Unfortuntely, this part of the data cannot be used to store any type of Os structures, only private data. But still. The corresponding concept is a "MMUContext", and again, it is provided by the mmu.library. A program thus can setup a context, shares parts of the global (default) context with the Os structure with other programs, but protect its private data from other programs running in other contexts. Again, the concept is there, the software that provides such magic is there (including context switches, context creation and administration) just that nobody uses it for that. Private memory is that - private. It should not have MEMF_PUBLIC set, it does not exist in exec, but the memory.library can provide such memory. The memory.library *also* exists, just no programs using it.
4) Is virtual memory possible? Not for Os structures. Only for private memory. Reasons are basically that swapping in memory requires a task switch, thus breaks any Forbid() and Disable(), and the Os is not prepared for that. However, if a program uses private memory (via the memory.library) such memory *can* be swapped out, and loaded from disk (automatically), again - of course - breaking the Forbid() state. Just that there are no programs using the mechanism. Thus, a lot of plans and a lot of mechanisms have been created, but since the software basis and the user basis broke away, all the mechanisms and constructions remained unused.
-
3) Can programs still isolate *parts* of their data? Yes, they can. Unfortuntely, this part of the data cannot be used to store any type of Os structures, only private data. But still.
...
4) Is virtual memory possible? Not for Os structures. Only for private memory. Reasons are basically that swapping in memory requires a task switch, thus breaks any Forbid() and Disable(), and the Os is not prepared for that.
I also think most of these restrictions can be removed by doing the proper changes in the OS and most of the OS can be MP also.
-
I have to admit I am no SELinux expert (I just disable it) but what I had in mind when I wrote that is that you use shared memory if two processes want common memory access and SELinux is used to restrict access right to this shared memory.
Shared memory like almost everything else in UNIX works like files and the standard access controls exist there.
SELinux exists for finer grained control over stuff and then you get into the capabilities a process should be allowed to used etc. It's mainly about stopping someone getting into a process and then using it to take over the rest of the machine by limiting what a process can to do down to exactly what it needs.
If you want something simpler than SELinux there are 3 or so other systems that can be used. I personally like TOMOYO.
-
If it's fine grained enough that you can protect the active process then it's basically an MMU.
I would disagree there. You could have a protection unit that could generate exceptions based on ranges that are selectable in byte resolution and it wouldn't be an MMU.
-
The difference is that computers with MMU can use a static memory position as opposed when you actually need the relocation table.
If you don't want an MMU but you want to link everything at the same address you could use bank switching. I.e. you allocate n x tasksize regions to hold tasks. When you want to switch task you change a bank register that moves the new task into the fixed address space. Obviously that's a lot less flexible than having a real MMU but if it's a toy system then it doesn't matter. Of course that means some hardware work.
I found the x86-32-ELF format more complicated than Amiga-68k-HUFF. I didn't have a look at µCLinux in that aspect.
I'm not an expert in the in's and out's of ELF by a long shot but I have written code to load m68k-elf executables into a simulator etc. For the most part ELF is just header + sub headers for blocks of data. I'm sure it gets more complicated once OS level stuff like libraries etc get involved but again if you're just playing around... If this is your own OS you don't have to do relocation in whatever way ELF does it. I don't think you mentioned a target but if it's m68k you could make you could generate flat m68k elf binaries that relocate themselves in the start up code before jumping into main(). Or just tell GCC to generate position independent code so it doesn't matter where it's loaded. I would take a look at uClinux's elf2bflt tool and see what they do to take an ELF and generate a simpler BFLT from it.[/QUOTE]
As for making use of MMU-less ARM (or perhaps MIPS). It's just noted that ELF decoding and relocation might be less straightforward than I thought. But most importantly is that I know gcc + ld may and will produce relocatable binaries.
What GCC generates will depend on the target. If it's a target that doesn't have an OS what type of relocatable binary would it generate?
And perhaps if a sufficient handleable QFP surface mount CPU with MMU can be had.
For ARM: AllWinner have some Cortex-A? parts in TQFP 144 IIRC. If you want something smaller then there are older Freescale i.MX parts that are ARM9 based and are in TQFP 100 I think.
-
4) Is virtual memory possible? Not for Os structures. Only for private memory. Reasons are basically that swapping in memory requires a task switch, thus breaks any Forbid() and Disable(), and the Os is not prepared for that. However, if a program uses private memory (via the memory.library) such memory *can* be swapped out, and loaded from disk (automatically), again - of course - breaking the Forbid() state. Just that there are no programs using the mechanism. Thus, a lot of plans and a lot of mechanisms have been created, but since the software basis and the user basis broke away, all the mechanisms and constructions remained unused.
Long. long ago (end of 90s I believe) I was thinking of using memory.library to allocate private memory that could be swapped away. But I realized that due to access restrictions it is not safe to pass private memory to any system call. Another problem that if I was using swappable memory it could work fine on my machine even if I violated access restrictions. On some another machine low on memory it could crash because memory was swapped away. And last and maybe also least reason: it would give nothing to WinUAE users...
So I scrapped that idea.
-
The main obstacle to get some Unix style environment is to be able to load arbitrary code into arbitrary locations. So I remembered that the 68000 lacked MMU but still provided arbitrary program loads and multitasking. But never investigated in depth how this was done.
The Amiga Hunk format uses a relocatable loader. It's convenient and more efficient in some cases but not necessary for the 68000 which can only do 16 bit PC relative addressing (plus or minus 32kB). It's possible to create larger position independent code with multiple branches and other means. The 68020+ has 32 bit branches and supports 32 bit offsets for full 32 bit PC relative addressing and unlimited size position independent code within it's addressable range while being reasonably efficient. It would not be efficient for a 64 bit (addressing) processor to provide for fully independent code through PC relative addressing as the offsets would grow too large. Programs aren't that big anyway. Each processor will have a different limit for the maximum position independent PC relative code and it may not match the 68k, especially with 32 bit fixed length RISC processor encodings which do not have encoding space for 32 bit offsets. The 68020+ is one of the easiest processors to create large position independent code for.
This insight of AmigaOS-68k and MMU less processors now means I know how to make this happen in other environments. Decoding ELF binaries with relocation table and making use of it seems however way more complicated than the Amiga Huff format. The loader has to copy the ELF binary code into memory and then find the relocation table and interpret it. This could form an self hosting ARM environment that works similar to AmigaOS 68k minus the custom graphics and sound.
Relocation like the Amiga Hunk format uses really isn't that complicated. Basic ELF or Amiga Hunk format support including relocation is not difficult. Supporting all the variations of hunk types in a robust way is more time consuming.
-
The target is ARM or MIPS and only the type of relocation table (list) that my compiler produce will be supported. ;)
Dunno if GCC/LLVM will produce PC relative code for ARM/MIPS though. m68k is not particular big in embedded for new designs..
-
The target is ARM or MIPS and only the type of relocation table (list) that my compiler produce will be supported. ;)
It looks like 32 bit ARM branches allow a signed 24 bit offset for forward and backward branches of +-32MB which is very good (conditional and subroutine branches have the same range as all branch instructions include a condition field). Thumb only allows 8 bit offsets for conditional branches (+-256 bytes), 11 bits for unconditional offsets (+-2KB) and 2 instructions for subroutines giving +-4MB which is awful for trying to produce position independent code (PIC). ARM v8/AARCH 64 (new 64 bit ARM) has a PC relative range of +-1MB for conditional branches (and literal loads/stores), unconditional and subroutines allow a range of +-128MB. There are other instructions that are helpful for PIC besides branches but they are the most important and they put a limit on the maximum size of PIC.
In comparison, the 68000 allows 16 bit offsets for all (implicit) PC relative branches giving +-32kB of range. The 68020 allows 32 bit offsets for all branches giving +-4GB for branches and has support for 32 bit offsets for all addressing mode types. It's easier to make a large PIC program for the 68020 than it is for any ARM processor.
Dunno if GCC/LLVM will produce PC relative code for ARM/MIPS though. m68k is not particular big in embedded for new designs..
I believe GCC can be told to produce PC relative (PIC) code with the -fPIC and -fPIE options. However, the code probably has to be written in a particular way to produce a position independent executable (PIE). Do a search for writing re-entrant code for the details. Amiga libraries require re-entrant code with the same restrictions. All support functions have to be re-entrant as well and may require linking with different support libraries.
The 68k is not popular for embedded because there aren't very many options or powerful 68k processors currently available. If an fpga 68k processor with 68060 like performance is good enough for your embedded needs then let me know. I might know where to find one ;).
-
Code that is given a different pointer to input and scratch variable space for each instance run ought to also be re-entrant code?
An FPGA would be more complicated and require a little bit to much standby current. FPGA Replay has a quite fast m68k CPU alike clone.
-
Code that is given a different pointer to input and scratch variable space for each instance run ought to also be re-entrant code?
That's right. Global/static variables other than constants generally can't be shared by separate instances of tasks/processes using shared re-entrant code. They are usually allocated on the stack or dynamically on the Amiga. It should be possible for a loader to allocate certain designated hunks for each re-entrant task/process that is started but, as far as I know, the Amiga doesn't support this. It would have made it simpler to write re-entrant programs and have them supported by compilers. It would be possible to require position independent code without allowing for multiple simultaneous tasks/processes but it makes sense to share code if going to the trouble of making it position independent.
An FPGA would be more complicated and require a little bit to much standby current. FPGA Replay has a quite fast m68k CPU alike clone.
An fpga gives flexibility with all programmable inputs and outputs in a one chip solution *if* the processing power of a hard chip is not needed. Of course there are hard ARM processors available in fpgas but that is more expensive. I don't know about the current draw. The fpga power usage seems tiny to me but it's all relative when talking about embedded applications. The TG68 in the fpga Arcade/Replay is not particularly fast but it requires very few resources, it's fully 68020 compatible, it's mostly debugged, it has a friendly license and it should give fairly predictable and consistent performance. It can give 68030 to 68040 performance depending on the speed, resources and configuration in the fpga. The Phoenix/Apollo fpga 68k CPU is coming along and should be available in a mostly debugged state in the next 6 months. Performance ranging from 68040 to 68060+ should be possible but more fpga resources are required than the TG68. The only hard modern 68k processor worth considering is Innovasic's FIDO.
http://www.innovasic.com/Products/fido1100-communication-controller
It's an innovative design that would be awesome for some applications and not so good for others. It's a CPU32 (between 68000 and 68020 with full 32 bit branch and PC relative support) ISA at only 66MHz but then it gives very consistent performance with lower power consumption possible by using a lower clock speed. The 68k can be very powerful with a low clock speed due to efficient memory performance (ARM's load/store architecture is at a disadvantage working in memory without OoO).
-
It should be possible for a loader to allocate certain designated hunks for each re-entrant task/process that is started but, as far as I know, the Amiga doesn't support this.
The dos.library loader does not support this indeed, basically because its job is over once the code is loaded, and then it is too late. However, there are options for the SAS/C which do exactly that: Put all constants into the CODE (text) segment, and allocate all globals (DATA) manually when starting up the program. Globals are then accessed indirectly through (a4), and a4 is set to a dynamically allocated piece of memory each time the program is run, so code is "pure" and can be made resident.
-
How did that "PURE (http://www.jaruzel.com/projects/amigados-guide-help/RESIDENT.html)" executables work in relation to this?
-
The dos.library loader does not support this indeed, basically because its job is over once the code is loaded, and then it is too late. However, there are options for the SAS/C which do exactly that: Put all constants into the CODE (text) segment, and allocate all globals (DATA) manually when starting up the program. Globals are then accessed indirectly through (a4), and a4 is set to a dynamically allocated piece of memory each time the program is run, so code is "pure" and can be made resident.
My memory was a little fuzzy about how this works. The SAS/C option is convenient as it allows programs that weren't designed to be re-entrant to work with some caveats (+-32kB of global data addressing similar to Small Data as I recall). I once considered adding similar support to vbcc but it was so low of priority that I haven't got around to trying it.
How did that "PURE (http://www.jaruzel.com/projects/amigados-guide-help/RESIDENT.html)" executables work in relation to this?
Executables that are "pure" are re-entrant (by other tasks/processes) and share code (usually data constants also). The pure file protection bit can be set on any program and the AmigaOS will treat it as a re-entrant program but it will likely crash. Libraries are required to be pure/re-entrant but don't require the pure file protection bit to be set. The relocatable loader is used the first time a pure program is loaded so AmigaOS does not require pure programs to be position independent. However, absolute 32 bit addressing (commonly used for global variables of large programs) which is relocated is less useful with pure programs.
-
This must mean that programs that has the PURE flag is feed with a parameter that tells it where this run of the code has it's variables etc.
-
This must mean that programs that has the PURE flag is feed with a parameter that tells it where this run of the code has it's variables etc.
Pure flag only indicates this program can be made resident (preloaded to ram with resident command). Internally programs can have different means to achieve re-entrancy. It could be semaphore or dynamically allocate ram for each instance.
SAS/C and GCC which have built-in support to build re-entrant (pure) code. They use custom startup code to copy data segment to newly allocated space. Pointer to data segment is placed to a4 and code is compiled to address variables relative to a4. Some special care is needed by a programmer, though.
You can of course choose to develop your program to be re-entrant without using any compiler support but then you can't use many c library calls.
-
Pure flag only indicates this program can be made resident (preloaded to ram with resident command). Internally programs can have different means to achieve re-entrancy. It could be semaphore or dynamically allocate ram for each instance.
Code initialization may be different for a pure program but I believe the AmigaOS (dos.library?) has code that maintains a list of currently used pure programs (resident list?), allows for finding and reusing the code and flushes the code under certain conditions. ThoR probably knows all the details of how it works. The AmigaOS does have the Resident CLI command to make pure programs resident (it can also show the use count of resident commands) and the H (Hold not Hidden) file attribute flag to make them resident when first used (requires the Pure bit also) without using the Resident command (saves a few lines in the S:Startup-Sequence for making Assign, Mount and Execute resident for example). There appears to be automagic support from the AmigaOS.
-
I would disagree there. You could have a protection unit that could generate exceptions based on ranges that are selectable in byte resolution and it wouldn't be an MMU.
Why wouldn't it be?
SAS/C and GCC which have built-in support to build re-entrant (pure) code. They use custom startup code to copy data segment to newly allocated space. Pointer to data segment is placed to a4 and code is compiled to address variables relative to a4. Some special care is needed by a programmer, though.
I don't know if they also store a4 in struct Task tc_UserData, but that is another way of doing it.
-
How did that "PURE (http://www.jaruzel.com/projects/amigados-guide-help/RESIDENT.html)" executables work in relation to this?
First, the PURE flag is just a pure indicator bit (no pun intended) for the resident program that is tested; trying to make a program resident with the pure flag cleared creates an error, but the resident program does actually not check for pure-ness in any other way. The ARP/Software distillery version of it had at least a checksum they tested for, but CBM removed that logic (but otherwise used lots of code from the software distillery to replace the tripos/BCPL code).
How other resident programs work is really program dependent. I personally avoid using global memory for such programs and put everything on the stack; there is no higher magic otherwise. Oh yes, probably avoid using code from the standard library. memcpy/strcpy and frieds are fine, but high-level stuff like printf()/fopen() are not. Use the dos PrintF() instead, for example.