How can executables work when being thrown into different addresses on 68000 ?

[Fats]

3) Can programs still isolate *parts* of their data? Yes, they can. Unfortuntely, this part of the data cannot be used to store any type of Os structures, only private data. But still.
...

4) Is virtual memory possible? Not for Os structures. Only for private memory. Reasons are basically that swapping in memory requires a task switch, thus breaks any Forbid() and Disable(), and the Os is not prepared for that.

I also think most of these restrictions can be removed by doing the proper changes in the OS and most of the OS can be MP also.

[donpalmera]

I have to admit I am no SELinux expert (I just disable it) but what I had in mind when I wrote that is that you use shared memory if two processes want common memory access and SELinux is used to restrict access right to this shared memory.

Shared memory like almost everything else in UNIX works like files and the standard access controls exist there.
SELinux exists for finer grained control over stuff and then you get into the capabilities a process should be allowed to used etc. It's mainly about stopping someone getting into a process and then using it to take over the rest of the machine by limiting what a process can to do down to exactly what it needs.
If you want something simpler than SELinux there are 3 or so other systems that can be used. I personally like TOMOYO.

[donpalmera]

If it's fine grained enough that you can protect the active process then it's basically an MMU.

I would disagree there. You could have a protection unit that could generate exceptions based on ranges that are selectable in byte resolution and it wouldn't be an MMU.

[donpalmera]

The difference is that computers with MMU can use a static memory position as opposed when you actually need the relocation table.

If you don't want an MMU but you want to link everything at the same address you could use bank switching. I.e. you allocate n x tasksize regions to hold tasks. When you want to switch task you change a bank register that moves the new task into the fixed address space. Obviously that's a lot less flexible than having a real MMU but if it's a toy system then it doesn't matter. Of course that means some hardware work.

I found the x86-32-ELF format more complicated than Amiga-68k-HUFF. I didn't have a look at µCLinux in that aspect.

I'm not an expert in the in's and out's of ELF by a long shot but I have written code to load m68k-elf executables into a simulator etc. For the most part ELF is just header + sub headers for blocks of data. I'm sure it gets more complicated once OS level stuff like libraries etc get involved but again if you're just playing around... If this is your own OS you don't have to do relocation in whatever way ELF does it. I don't think you mentioned a target but if it's m68k you could make you could generate flat m68k elf binaries that relocate themselves in the start up code before jumping into main(). Or just tell GCC to generate position independent code so it doesn't matter where it's loaded. I would take a look at uClinux's elf2bflt tool and see what they do to take an ELF and generate a simpler BFLT from it.[/QUOTE]

As for making use of MMU-less ARM (or perhaps MIPS). It's just noted that ELF decoding and relocation might be less straightforward than I thought. But most importantly is that I know gcc + ld may and will produce relocatable binaries.

What GCC generates will depend on the target. If it's a target that doesn't have an OS what type of relocatable binary would it generate?

And perhaps if a sufficient handleable QFP surface mount CPU with MMU can be had.

For ARM: AllWinner have some Cortex-A? parts in TQFP 144 IIRC. If you want something smaller then there are older Freescale i.MX parts that are ARM9 based and are in TQFP 100 I think.

[itix]

4) Is virtual memory possible? Not for Os structures. Only for private memory. Reasons are basically that swapping in memory requires a task switch, thus breaks any Forbid() and Disable(), and the Os is not prepared for that. However, if a program uses private memory (via the memory.library) such memory *can* be swapped out, and loaded from disk (automatically), again - of course - breaking the Forbid() state. Just that there are no programs using the mechanism.  Thus, a lot of plans and a lot of mechanisms have been created, but since the software basis and the user basis broke away, all the mechanisms and constructions remained unused.

Long. long ago (end of 90s I believe) I was thinking of using memory.library to allocate private memory that could be swapped away. But I realized that due to access restrictions it is not safe to pass private memory to any system call. Another problem that if I was using swappable memory it could work fine on my machine even if I violated access restrictions. On some another machine low on memory it could crash because memory was swapped away. And last and maybe also least reason: it would give nothing to WinUAE users...

So I scrapped that idea.

[matthey]

The main obstacle to get some Unix style environment is to be able to load arbitrary code into arbitrary locations. So I remembered that the 68000 lacked MMU but still provided arbitrary program loads and multitasking. But never investigated in depth how this was done.

The Amiga Hunk format uses a relocatable loader. It's convenient and more efficient in some cases but not necessary for the 68000 which can only do 16 bit PC relative addressing (plus or minus 32kB). It's possible to create larger position independent code with multiple branches and other means. The 68020+ has 32 bit branches and supports 32 bit offsets for full 32 bit PC relative addressing and unlimited size position independent code within it's addressable range while being reasonably efficient. It would not be efficient for a 64 bit (addressing) processor to provide for fully independent code through PC relative addressing as the offsets would grow too large. Programs aren't that big anyway. Each processor will have a different limit for the maximum position independent PC relative code and it may not match the 68k, especially with 32 bit fixed length RISC processor encodings which do not have encoding space for 32 bit offsets. The 68020+ is one of the easiest processors to create large position independent code for.

This insight of AmigaOS-68k and MMU less processors now means I know how to make this happen in other environments. Decoding ELF binaries with relocation table and making use of it seems however way more complicated than the Amiga Huff format. The loader has to copy the ELF binary code into memory and then find the relocation table and interpret it. This could form an self hosting ARM environment that works similar to AmigaOS 68k minus the custom graphics and sound.

Relocation like the Amiga Hunk format uses really isn't that complicated. Basic ELF or Amiga Hunk format support including relocation is not difficult. Supporting all the variations of hunk types in a robust way is more time consuming.

[freqmax]

The target is ARM or MIPS and only the type of relocation table (list) that my compiler produce will be supported.

Dunno if GCC/LLVM will produce PC relative code for ARM/MIPS though. m68k is not particular big in embedded for new designs..

[matthey]

The target is ARM or MIPS and only the type of relocation table (list) that my compiler produce will be supported.

It looks like 32 bit ARM branches allow a signed 24 bit offset for forward and backward branches of +-32MB which is very good (conditional and subroutine branches have the same range as all branch instructions include a condition field). Thumb only allows 8 bit offsets for conditional branches (+-256 bytes), 11 bits for unconditional offsets (+-2KB) and 2 instructions for subroutines giving +-4MB which is awful for trying to produce position independent code (PIC). ARM v8/AARCH 64 (new 64 bit ARM) has a PC relative range of +-1MB for conditional branches (and literal loads/stores), unconditional and subroutines allow a range of +-128MB. There are other instructions that are helpful for PIC besides branches but they are the most important and they put a limit on the maximum size of PIC.

In comparison, the 68000 allows 16 bit offsets for all (implicit) PC relative branches giving +-32kB of range. The 68020 allows 32 bit offsets for all branches giving +-4GB for branches and has support for 32 bit offsets for all addressing mode types. It's easier to make a large PIC program for the 68020 than it is for any ARM processor.

Dunno if GCC/LLVM will produce PC relative code for ARM/MIPS though. m68k is not particular big in embedded for new designs..

I believe GCC can be told to produce PC relative (PIC) code with the -fPIC and -fPIE options. However, the code probably has to be written in a particular way to produce a position independent executable (PIE). Do a search for writing re-entrant code for the details. Amiga libraries require re-entrant code with the same restrictions. All support functions have to be re-entrant as well and may require linking with different support libraries.

The 68k is not popular for embedded because there aren't very many options or powerful 68k processors currently available. If an fpga 68k processor with 68060 like performance is good enough for your embedded needs then let me know. I might know where to find one .

[freqmax]

Code that is given a different pointer to input and scratch variable space for each instance run ought to also be re-entrant code?

An FPGA would be more complicated and require a little bit to much standby current. FPGA Replay has a quite fast m68k CPU alike clone.

[matthey]

Code that is given a different pointer to input and scratch variable space for each instance run ought to also be re-entrant code?

That's right. Global/static variables other than constants generally can't be shared by separate instances of tasks/processes using shared re-entrant code. They are usually allocated on the stack or dynamically on the Amiga. It should be possible for a loader to allocate certain designated hunks for each re-entrant task/process that is started but, as far as I know, the Amiga doesn't support this. It would have made it simpler to write re-entrant programs and have them supported by compilers. It would be possible to require position independent code without allowing for multiple simultaneous tasks/processes but it makes sense to share code if going to the trouble of making it position independent.

An FPGA would be more complicated and require a little bit to much standby current. FPGA Replay has a quite fast m68k CPU alike clone.

An fpga gives flexibility with all programmable inputs and outputs in a one chip solution *if* the processing power of a hard chip is not needed. Of course there are hard ARM processors available in fpgas but that is more expensive. I don't know about the current draw. The fpga power usage seems tiny to me but it's all relative when talking about embedded applications. The TG68 in the fpga Arcade/Replay is not particularly fast but it requires very few resources, it's fully 68020 compatible, it's mostly debugged, it has a friendly license and it should give fairly predictable and consistent performance. It can give 68030 to 68040 performance depending on the speed, resources and configuration in the fpga. The Phoenix/Apollo fpga 68k CPU is coming along and should be available in a mostly debugged state in the next 6 months. Performance ranging from 68040 to 68060+ should be possible but more fpga resources are required than the TG68. The only hard modern 68k processor worth considering is Innovasic's FIDO.

http://www.innovasic.com/Products/fido1100-communication-controller

It's an innovative design that would be awesome for some applications and not so good for others. It's a CPU32 (between 68000 and 68020 with full 32 bit branch and PC relative support) ISA at only 66MHz but then it gives very consistent performance with lower power consumption possible by using a lower clock speed. The 68k can be very powerful with a low clock speed due to efficient memory performance (ARM's load/store architecture is at a disadvantage working in memory without OoO).

[guest11527]

It should be possible for a loader to allocate certain designated hunks for each re-entrant task/process that is started but, as far as I know, the Amiga doesn't support this.

The dos.library loader does not support this indeed, basically because its job is over once the code is loaded, and then it is too late. However, there are options for the SAS/C which do exactly that: Put all constants into the CODE (text) segment, and allocate all globals (DATA) manually when starting up the program. Globals are then accessed indirectly through (a4), and a4 is set to a dynamically allocated piece of memory each time the program is run, so code is "pure" and can be made resident.

[freqmax]

How did that "PURE" executables work in relation to this?

[matthey]

The dos.library loader does not support this indeed, basically because its job is over once the code is loaded, and then it is too late. However, there are options for the SAS/C which do exactly that: Put all constants into the CODE (text) segment, and allocate all globals (DATA) manually when starting up the program. Globals are then accessed indirectly through (a4), and a4 is set to a dynamically allocated piece of memory each time the program is run, so code is "pure" and can be made resident.

My memory was a little fuzzy about how this works. The SAS/C option is convenient as it allows programs that weren't designed to be re-entrant to work with some caveats (+-32kB of global data addressing similar to Small Data as I recall). I once considered adding similar support to vbcc but it was so low of priority that I haven't got around to trying it.

How did that "PURE" executables work in relation to this?

Executables that are "pure" are re-entrant (by other tasks/processes) and share code (usually data constants also). The pure file protection bit can be set on any program and the AmigaOS will treat it as a re-entrant program but it will likely crash. Libraries are required to be pure/re-entrant but don't require the pure file protection bit to be set. The relocatable loader is used the first time a pure program is loaded so AmigaOS does not require pure programs to be position independent. However, absolute 32 bit addressing (commonly used for global variables of large programs) which is relocated is less useful with pure programs.

[freqmax]

This must mean that programs that has the PURE flag is feed with a parameter that tells it where this run of the code has it's variables etc.

[itix]

This must mean that programs that has the PURE flag is feed with a parameter that tells it where this run of the code has it's variables etc.

Pure flag only indicates this program can be made resident (preloaded to ram with resident command). Internally programs can have different means to achieve re-entrancy. It could be semaphore or dynamically allocate ram for each instance.

SAS/C and GCC which have built-in support to build re-entrant (pure) code. They use custom startup code to copy data segment to newly allocated space. Pointer to data segment is placed to a4 and code is compiled to address variables relative to a4. Some special care is needed by a programmer, though.

You can of course choose to develop your program to be re-entrant without using any compiler support but then you can't use many c library calls.

[matthey]

Pure flag only indicates this program can be made resident (preloaded to ram with resident command). Internally programs can have different means to achieve re-entrancy. It could be semaphore or dynamically allocate ram for each instance.

Code initialization may be different for a pure program but I believe the AmigaOS (dos.library?) has code that maintains a list of currently used pure programs (resident list?), allows for finding and reusing the code and flushes the code under certain conditions. ThoR probably knows all the details of how it works. The AmigaOS does have the Resident CLI command to make pure programs resident (it can also show the use count of resident commands) and the H (Hold not Hidden) file attribute flag to make them resident when first used (requires the Pure bit also) without using the Resident command (saves a few lines in the S:Startup-Sequence for making Assign, Mount and Execute resident for example). There appears to be automagic support from the AmigaOS.