FS: Amiga Guru Book by Ralph Babel

[bloodline]

This seems to somehow confuse load files with object files and libraries. Oh well. For this particular topic, the Bantam book is good enough.

This website does seem to be very confusing to me... But I put that down to my lack of understanding.

[nicholas]

This seems to somehow confuse load files with object files and libraries. Oh well. For this particular topic, the Bantam book is good enough.

Which I also have if you want to borrow it Matt.

[bloodline]

Which I also have if you want to borrow it Matt.

I think I will have to take you up on the offer!

Please see which book has the best description of the Hunk format and how Loadseg works, then PM your PayPal address and I'll send over the cost of shipping (plus any other overheads) cheers

[paul1981]

I think I will have to take you up on the offer!

Please see which book has the best description of the Hunk format and how Loadseg works, then PM your PayPal address and I'll send over the cost of shipping (plus any other overheads) cheers

You should have bought the GURU book before he changed his mind.

[guest11527]

Please see which book has the best description of the Hunk format and how Loadseg works

How much detail do you need? If we can disregard overlay files (rarely used), then LoadSeg() is really rather trivial. A binary load file consists of a HUNK_HEADER, which contains the number of hunks, and for each hunk, the number of long words to allocate per hunk, and the memory type to use in its upper two bits.

Following that is a sequence of HUNK_CODE, HUNK_DATA or HUNK_BSS, each of which may be followed by one or multiple relocation information (HUNK_RELOC32 or HUNK_RELOC32SHORT). HUNK_CODE and HUNK_DATA contains both the number of longwords contained in the hunk (which may be less than the number of LWs allocated in the header), followed by the data itself. HUNK_BSS is just blank space, hence no data included.

HUNK_RELOC32 defines which offsets in the hunk loaded before have to be relocated, and the base address of which hunk has to be used for relocation. HUNK_RELOC32 consists of one or multiple lists, each list starts with the number of entries to relocate, followed by the hunk to be used as base address, followed by the offsets in the just loaded hunk that have to be relocated, one longword per entry. All LoadSeg() does is to add the base address of the hunk indicated in the second long word of the list to the long word at the base address of the current hunk plus the offset in the list.

If the number of entries is zero, then HUNK_RELOC32 ends.

HUNK_RELOC32SHORT is similar, except that the offsets are 16 bit wide, not 32bit wide.

HUNK_END ends a hunk.

Hence, if you want to define a syntax:

Code: [Select]

binary load file := HUNK_HEADER HUNKS
HUNKS := one or multiple HUNK
HUNK  := HUNK_BODY, HUNK_RELOCs, HUNK_END
HUNK_BODY := HUNK_CODE or HUNK_DATA or HUNK_BSS
HUNK_RELOCs := one or multiple HUNK_RELOC32 or HUNK_RELOC32SHORT
HUNK_RELOC32 := 0x3ec RELOC32TABLES
RELOC32TABLES := END_TABLE or FULL_LONG_TABLE
END_TABLE := 0
FULL_LONG_TABLE := count, base_hunk, count times 32bitoffsets
HUNK_RELOC32SHORT := 0x3fc RELOC32SHORTTABLES
RELOC32SHORTTABLES := END_TABLE or FULL_SHORT_TABLE
FULL_SHORT_TABLE := count, base_hunk, count times 16bitoffsets, cludgefill
where "cludgefill" is an empty 16 bit word if the number of entries is odd, i.e. does not end on a LW boundary. All other entries are longwords (BCPL cells).

Relocation works as such:

longword at (address of(current hunkt)+ 32/16bitoffset) += address of(base hunk)

Things get a bit more hairy for overlay files, but it does not require any other types except HUNK_BREAK or HUNK_OVERLAY. All the other hunk types in the above web resource are reserved for libraries or object files, and you do not need them.

To work on binary load files, I suggest:

http://aminet.net/package/dev/misc/Hunk

[bloodline]

How much detail do you need? If we can disregard overlay files (rarely used), then LoadSeg() is really rather trivial. A binary load file consists of a HUNK_HEADER, which contains the number of hunks, and for each hunk, the number of long words to allocate per hunk, and the memory type to use in its upper two bits.

I want to be able to parse the Hunk executable file, and load it into an arbitrary memory location, setting up sufficient heap and stack to point my 68k emulator at it and execute what it finds.

In my current implementation, I have allocated 16megabytes for use by the emulator. I wish to load the file at address 0x200000 (I have 8megabytes reserved at this location for code), relocating the code and allocating memory  as required.

I have a pointer at address 0x000004 pointing to a faked exec structure (currently located at 0xafe7fe, which seems to be away from any important address space), with the jump table set up (as defined by Christian Vogelgsang for his VAMOS project):

    my fake lib entry:
real amiga lib entry:

So I can trap exec library calls.

The executable I am using as my test is the AmigaDOS 1.3 command "Type", perhaps not the best example, but it is small and should only call the exec.library and dos.library.

(I will be ignoring overlay hunks).

Following that is a sequence of HUNK_CODE, HUNK_DATA or HUNK_BSS, each of which may be followed by one or multiple relocation information (HUNK_RELOC32 or HUNK_RELOC32SHORT). HUNK_CODE and HUNK_DATA contains both the number of longwords contained in the hunk (which may be less than the number of LWs allocated in the header), followed by the data itself. HUNK_BSS is just blank space, hence no data included.

HUNK_RELOC32 defines which offsets in the hunk loaded before have to be relocated, and the base address of which hunk has to be used for relocation. HUNK_RELOC32 consists of one or multiple lists, each list starts with the number of entries to relocate, followed by the hunk to be used as base address, followed by the offsets in the just loaded hunk that have to be relocated, one longword per entry. All LoadSeg() does is to add the base address of the hunk indicated in the second long word of the list to the long word at the base address of the current hunk plus the offset in the list.

If the number of entries is zero, then HUNK_RELOC32 ends.

HUNK_RELOC32SHORT is similar, except that the offsets are 16 bit wide, not 32bit wide.

HUNK_END ends a hunk.

Hence, if you want to define a syntax:

Code: [Select]

binary load file := HUNK_HEADER HUNKS
HUNKS := one or multiple HUNK
HUNK  := HUNK_BODY, HUNK_RELOCs, HUNK_END
HUNK_BODY := HUNK_CODE or HUNK_DATA or HUNK_BSS
HUNK_RELOCs := one or multiple HUNK_RELOC32 or HUNK_RELOC32SHORT
HUNK_RELOC32 := 0x3ec RELOC32TABLES
RELOC32TABLES := END_TABLE or FULL_LONG_TABLE
END_TABLE := 0
FULL_LONG_TABLE := count, base_hunk, count times 32bitoffsets
HUNK_RELOC32SHORT := 0x3fc RELOC32SHORTTABLES
RELOC32SHORTTABLES := END_TABLE or FULL_SHORT_TABLE
FULL_SHORT_TABLE := count, base_hunk, count times 16bitoffsets, cludgefill
where "cludgefill" is an empty 16 bit word if the number of entries is odd, i.e. does not end on a LW boundary. All other entries are longwords (BCPL cells).

Relocation works as such:

longword at (address of(current hunkt)+ 32/16bitoffset) += address of(base hunk)

Things get a bit more hairy for overlay files, but it does not require any other types except HUNK_BREAK or HUNK_OVERLAY. All the other hunk types in the above web resource are reserved for libraries or object files, and you do not need them.

To work on binary load files, I suggest:

http://aminet.net/package/dev/misc/Hunk

I think you have provided enough information for me to progress, I would appreciate a worked example if you have one... I appreciate this might not be the best place for that

[Pat the Cat]

...So I can trap exec library calls.

The executable I am using as my test is the AmigaDOS 1.3 command "Type", perhaps not the best example, but it is small and should only call the exec.library and dos.library.

(I will be ignoring overlay hunks).
...

I think you might be on to something there. I did try hacking into some 1.3 commands once with a cartridge dissassembler, single stepping etc, and was completely baffled by the amount of system activity going on for just very simple commands.

Your approach seems much more level headed... it was like a spaghetti waterfall, trying to track what on earth the processor was doing or which bit of ROM it was looping around. :confused:

And I certainly didn't have a clue how headers worked, that was an education. :rtfm: I guess largely designed 32 bit from very early in AmigaOS. Made adding fully 32bit CPUs a lot easier.

[guest11527]

I have a pointer at address 0x000004 pointing to a faked exec structure (currently located at 0xafe7fe, which seems to be away from any important address space), with the jump table set up (as defined by Christian Vogelgsang for his VAMOS project):

But you are aware that vamos already has a segment loader? You find it in amitools/binfmt/load_image(). In particular, amitools/binfmt/hunk/HunkLoadSegFile.py also includes the parser of the Hunk format.

It is certainly a bit convoluted and probably over-engineered, but it works. It shouldn't be too much of a problem to hack it up a little bit to include loading a binary to a specific address in VAMOS' address space.

The executable I am using as my test is the AmigaDOS 1.3 command "Type", perhaps not the best example, but it is small and should only call the exec.library and dos.library.

VAMOS is certainly capable of executing "type", this will work out of the box.

I think you have provided enough information for me to progress, I would appreciate a worked example if you have one... I appreciate this might not be the best place for that

Well, see above for  a working example in python. I also have working examples in C, but since you're more into vamos anyhow, I would simply take what it already offers.

[guest11527]

I think you might be on to something there. I did try hacking into some 1.3 commands once with a cartridge dissassembler, single stepping etc, and was completely baffled by the amount of system activity going on for just very simple commands.

Please note that 1.3 is a horse of a different color. The commands in C: of workbench 1.3 were written in BCPL, and hence used a lot of the BCPL magic. In particular, they depend on an *additional* structure within the code hunks that defines which "GlobVec" entries to populate. This structure has been phased out, and is not part of LoadSeg() or the hunk structure.

You find more about this obsolete junk (not hunk) here:

http://aminet.net/package/dev/misc/BCPL

Commands from 2.0 onwards were plain C, did not depend on any arcane BPCL magic, used C linkage and the plain HUNK format without the higher level structure that was defined by the Tripos loader.

[bloodline]

But you are aware that vamos already has a segment loader? You find it in amitools/binfmt/load_image(). In particular, amitools/binfmt/hunk/HunkLoadSegFile.py also includes the parser of the Hunk format.

I must confess, I've never seen python before, and I have trouble reading it. stumbling over Chris's code in a strange language isn't much fun.

It is certainly a bit convoluted and probably over-engineered, but it works. It shouldn't be too much of a problem to hack it up a little bit to include loading a binary to a specific address in VAMOS' address space.


VAMOS is certainly capable of executing "type", this will work out of the box.

That gives me hope

Well, see above for  a working example in python. I also have working examples in C, but since you're more into vamos anyhow, I would simply take what it already offers.

I would prefer C as that is the only language I have used in the past 17 years

[bloodline]

What I need really is to know what data expected to see at each offset.

at byte 0  I expect to see 0x3F3
at byte 4  I always see 0x0 (the spec says this should be "Strings")
at byte 8  I should see "table size" (what is the table? the total number of hunks +1?)
at byte 12 I should see the first hunk (I guess for a load file this should always be 0)
at byte 16 I should see the last hunk (so this tells me how many hunks I am expecting?)
at byte 20 I should get a list of hunk sizes? (I don't understand what this is)
at byte 24 not sure what I'm looking at...
at byte 28 I  see my first code hunk... (0x3E9)
at byte 32 Guess is the size of the code hunk in 4byte blocks?

[guest11527]

What I need really is to know what data expected to see at each offset.

at byte 0  I expect to see 0x3F3
at byte 4  I always see 0x0 (the spec says this should be "Strings")

This must be zero. It is used by some arcane BCPL linkage of libraries that is no longer supported (and which actually only worked with the dos.library anyhow.).

at byte 8  I should see "table size" (what is the table? the total number of hunks +1?)

This is just the number of hunks to load.

at byte 12 I should see the first hunk (I guess for a load file this should always be 0)

Yes.

at byte 16 I should see the last hunk (so this tells me how many hunks I am expecting?)

Yes. This is the number of hunks - 1. So for three hunks, this is 2. The exception are overlays.

at byte 20 I should get a list of hunk sizes? (I don't understand what this is)

What follows is for each hunk the number of long words to allocate for it. That is, if the binary consists of three hunks, three longwords are in this part of HUNK_HEADER. Each longword lists the number of long words to *allocate* for the hunk. I.e., you get the number of bytes to allocate by multiply the number in this section by four - or rightshift by 2.

Bits 30 and 31 are then shifted out. They contain the memory type. If both are zero, the type is MEMF_PUBLIC. If the bits are 01, it is MEMF_CHIP | MEMF_PUBLIC. If the bits are 10, it is MEMF_FAST | MEMF_PUBLIC. If it is 11, the next longword contains the memory type.

Actually, the combination 01 (MEMF_CHIP|MEMF_PUBLIC) is pretty much the only combination in use.

at byte 28 I  see my first code hunk... (0x3E9)
at byte 32 Guess is the size of the code hunk in 4byte blocks?

That is the number of longwords *to load*. Note that this can be smaller than the number of bytes to *allocate*. The remaining bytes remain undefined then. This allows, for example, to place the BSS data within another hunk, except that they are not cleared.

[bloodline]

Thomas!

Fantastic, very helpful! Many thanks. I will have more questions later

Also, I will proabably ignore the Memory flag high bits for this experiment... since I only have a single address space (for now )

[Pat the Cat]

Please note that 1.3 is a horse of a different color... Commands from 2.0 onwards were plain C, did not depend on any arcane BPCL magic, used C linkage and the plain HUNK format without the higher level structure that was defined by the Tripos loader.

Yes, AmigaOS started as a port. PL1 was a bit like that arcane magic stuff. Idea was you ran a programme in a sliced mainframe environment. Not stuff I ever really used, except to understand strictly defined data structures, pointers, other stuff.

Arrays have their uses, but for the finer definition of metrices and dimensional spaces, you definitely get advantage from more flexible means of defining data structures. I only realy experienced 1.2-3.1 Commodore era releases, much more the earlier. It all  started as BCPL and gradually got recoded into C, partly in the UK at first I think. Amiga INC did a lot of C coding, but just enough to make it the Amiga boot, and a lot of other people have recoded it since. Of course.

1.3 commands seem both more bulky and clunky than their later descendants. A couple paradoxically seem quicker, or perhaps that is just my imagination... I haven't witnessed enough of the later releases to comment really.

[guest11527]

Yes, AmigaOS started as a port.

Well, to some degree. Tripos was already 68K based, so it was more like putting Tripos on top of the (very small) micro-kernel exec had to offer. Actually, the number of changes are rather minimalistic, so the dos.library is really Tripos in action. Not much a port, but the real thing. Amiga genuine parts were really graphics (the big ugly) and intuition (a much better design than the rest), on which the workbench was put on top.

1.3 commands seem both more bulky and clunky than their later descendants. A couple paradoxically seem quicker, or perhaps that is just my imagination... I haven't witnessed enough of the later releases to comment really.

Well... compared to their Unix counterparts, even the Tripos commands were pretty short. This is because Tripos has a much better low-level support than Unix has. For example, command line parsing aka "ReadArgs()" and formatted priting ("WriteF()" - BCPLs "printf") were always part of Tripos, just not accessible from C, so a BCPL command already got a lot of services from the Os.

The only reason why the C counterparts are even shorter is a) the BCPL compiler did not exactly generate "ideal" code due to all the shifting of the BCPL "pointers" (not really, BCPL is a typeless language), and b) a lot of additional services were added to dos with v37, most of which came from the "arp" project of the software distillery, as for example the pattern matcher, and c) there was still a minimum amount of BCPL startup magic necessary within each command (the first hunk of the BCPL commands) that sets up the "GlobVec" of the command given the undocumented and arcane inner structure of the code hunks.

Thus, for example, while the average Unix C program uses "printf" from libc outside the kernel, and implements argument parsing itself, all this comes "for free" from the operating system, namely the dos.library, if you know how to use it.

[psxphill]

It all  started as BCPL and gradually got recoded into C, partly in the UK at first I think. Amiga INC did a lot of C coding, but just enough to make it the Amiga boot, and a lot of other people have recoded it since. Of course.

Amiga Corporation wrote all their code in C or assembler, when commodore bought them it was still pretty much smoke and mirrors. IIRC you loaded software onto the Amiga using a Sun workstation (the original Macintosh used a Lisa for a similar purpose http://www.folklore.org/StoryView.py?story=Shut_Up.txt).

They had a design for an operating system (called CAOS) but didn't have the time to do the development in house and they were let down by the people they contracted it to http://www.thule.no/haynie/caos.html So they had to start looking for existing solutions and TripOS was very similar to what they wanted and it would be easy to make it run on top of exec. TripOS was re-branded, a small amount of development was done and they had dos, a file system and commands.