FS: Amiga Guru Book by Ralph Babel

[bloodline]

Actually, it is a developer resource as well. It does list all the entries in the dos.library, the packet types, the CLI and Dos structures, overlays and the FFS structures. Just not in the detail necessary you would need for high quality development.

That's mostly because the AmigaDos manual is more or less a copy-paste job of the "Tripos Manual". The books are really almost identical (in the earlier editions at least) except a sed -e "s/Tripos/AmigaDos/", and some details. Structures in Tripos were mostly identical (CLI, for example, is exactly the same thing, the filing system is exactly the same thing, packets were identical, most commands were identical).

Archive.org seems to have a copy:

https://archive.org/details/1986-metacomco-intro-to-tripos

so you can judge yourself.

It is scary how much of the "Amiga" we know is actually Tripos!

[bloodline]

I've decided against selling it guys.

It still sits on the shelf alongside the other religious books.  

I might have to get your help understanding the Amiga Hunk format then

[bloodline]

You are most welcome to borrow the book if you want mate?

Very kind offer, but I am using:

http://amiga-dev.wikidot.com/file-format:hunk

which should do for now... I will see...

[bloodline]

This seems to somehow confuse load files with object files and libraries. Oh well. For this particular topic, the Bantam book is good enough.

This website does seem to be very confusing to me... But I put that down to my lack of understanding.

[bloodline]

Which I also have if you want to borrow it Matt.

I think I will have to take you up on the offer!

Please see which book has the best description of the Hunk format and how Loadseg works, then PM your PayPal address and I'll send over the cost of shipping (plus any other overheads) cheers

[bloodline]

How much detail do you need? If we can disregard overlay files (rarely used), then LoadSeg() is really rather trivial. A binary load file consists of a HUNK_HEADER, which contains the number of hunks, and for each hunk, the number of long words to allocate per hunk, and the memory type to use in its upper two bits.

I want to be able to parse the Hunk executable file, and load it into an arbitrary memory location, setting up sufficient heap and stack to point my 68k emulator at it and execute what it finds.

In my current implementation, I have allocated 16megabytes for use by the emulator. I wish to load the file at address 0x200000 (I have 8megabytes reserved at this location for code), relocating the code and allocating memory  as required.

I have a pointer at address 0x000004 pointing to a faked exec structure (currently located at 0xafe7fe, which seems to be away from any important address space), with the jump table set up (as defined by Christian Vogelgsang for his VAMOS project):

    my fake lib entry:
real amiga lib entry:

So I can trap exec library calls.

The executable I am using as my test is the AmigaDOS 1.3 command "Type", perhaps not the best example, but it is small and should only call the exec.library and dos.library.

(I will be ignoring overlay hunks).

Following that is a sequence of HUNK_CODE, HUNK_DATA or HUNK_BSS, each of which may be followed by one or multiple relocation information (HUNK_RELOC32 or HUNK_RELOC32SHORT). HUNK_CODE and HUNK_DATA contains both the number of longwords contained in the hunk (which may be less than the number of LWs allocated in the header), followed by the data itself. HUNK_BSS is just blank space, hence no data included.

HUNK_RELOC32 defines which offsets in the hunk loaded before have to be relocated, and the base address of which hunk has to be used for relocation. HUNK_RELOC32 consists of one or multiple lists, each list starts with the number of entries to relocate, followed by the hunk to be used as base address, followed by the offsets in the just loaded hunk that have to be relocated, one longword per entry. All LoadSeg() does is to add the base address of the hunk indicated in the second long word of the list to the long word at the base address of the current hunk plus the offset in the list.

If the number of entries is zero, then HUNK_RELOC32 ends.

HUNK_RELOC32SHORT is similar, except that the offsets are 16 bit wide, not 32bit wide.

HUNK_END ends a hunk.

Hence, if you want to define a syntax:

Code: [Select]

binary load file := HUNK_HEADER HUNKS
HUNKS := one or multiple HUNK
HUNK  := HUNK_BODY, HUNK_RELOCs, HUNK_END
HUNK_BODY := HUNK_CODE or HUNK_DATA or HUNK_BSS
HUNK_RELOCs := one or multiple HUNK_RELOC32 or HUNK_RELOC32SHORT
HUNK_RELOC32 := 0x3ec RELOC32TABLES
RELOC32TABLES := END_TABLE or FULL_LONG_TABLE
END_TABLE := 0
FULL_LONG_TABLE := count, base_hunk, count times 32bitoffsets
HUNK_RELOC32SHORT := 0x3fc RELOC32SHORTTABLES
RELOC32SHORTTABLES := END_TABLE or FULL_SHORT_TABLE
FULL_SHORT_TABLE := count, base_hunk, count times 16bitoffsets, cludgefill
where "cludgefill" is an empty 16 bit word if the number of entries is odd, i.e. does not end on a LW boundary. All other entries are longwords (BCPL cells).

Relocation works as such:

longword at (address of(current hunkt)+ 32/16bitoffset) += address of(base hunk)

Things get a bit more hairy for overlay files, but it does not require any other types except HUNK_BREAK or HUNK_OVERLAY. All the other hunk types in the above web resource are reserved for libraries or object files, and you do not need them.

To work on binary load files, I suggest:

http://aminet.net/package/dev/misc/Hunk

I think you have provided enough information for me to progress, I would appreciate a worked example if you have one... I appreciate this might not be the best place for that

[bloodline]

But you are aware that vamos already has a segment loader? You find it in amitools/binfmt/load_image(). In particular, amitools/binfmt/hunk/HunkLoadSegFile.py also includes the parser of the Hunk format.

I must confess, I've never seen python before, and I have trouble reading it. stumbling over Chris's code in a strange language isn't much fun.

It is certainly a bit convoluted and probably over-engineered, but it works. It shouldn't be too much of a problem to hack it up a little bit to include loading a binary to a specific address in VAMOS' address space.


VAMOS is certainly capable of executing "type", this will work out of the box.

That gives me hope

Well, see above for  a working example in python. I also have working examples in C, but since you're more into vamos anyhow, I would simply take what it already offers.

I would prefer C as that is the only language I have used in the past 17 years

[bloodline]

What I need really is to know what data expected to see at each offset.

at byte 0  I expect to see 0x3F3
at byte 4  I always see 0x0 (the spec says this should be "Strings")
at byte 8  I should see "table size" (what is the table? the total number of hunks +1?)
at byte 12 I should see the first hunk (I guess for a load file this should always be 0)
at byte 16 I should see the last hunk (so this tells me how many hunks I am expecting?)
at byte 20 I should get a list of hunk sizes? (I don't understand what this is)
at byte 24 not sure what I'm looking at...
at byte 28 I  see my first code hunk... (0x3E9)
at byte 32 Guess is the size of the code hunk in 4byte blocks?

[bloodline]

Thomas!

Fantastic, very helpful! Many thanks. I will have more questions later

Also, I will proabably ignore the Memory flag high bits for this experiment... since I only have a single address space (for now )

[bloodline]

Well after quite a bit of work, I did eventually figure out the Amiga Hunk file format. With all the executable files I tried, there are only three hunks I need to look out for: the Hunk header, which tells me how much ram to allocate (which in turn gives me my segment addresses), and the code Hunk and the data Hunk... everything else can be ignored, I found that my build of Deluxe Paint IV has a debug hunk for some reason.

Once you have found either a code or a data hunk, you need to load it into the previously allocated memory... then check if it has a reloc32 block (I haven't found any other type of reloc block) after it, if yes... then you need to put the addresses of the previously allocated segments at the listed offsets. and you keep scanning until you reach the end of the file.

None of this was making much sense, until I realised that the format is optimised to be processed while being streamed from a disk! So every operation you perform is done in the order you find it! Cli commands like "Type" only have a code hunk, no data or reloc32!

So now I can happily load an Amiga executable file, point my 68k at it and it will run!
I have built an exec.library Jump table style interface which when called, jumps execution out of the 68k emulator and into an x86 Obj-C compatible interface.

Which means I can write my libraries in Obj-C and their methods will be called by the 68k program (so much byte swapping!). I now have the arduous task of implementing the functions on demand. Exec is quite easy (I initially implemented AllocMem, FreeMem, openLibray and CloseLibrary), but dos is more difficult, because I never really used it before... but since I'm mostly testing cli programs, this is what I need to focus on for now.

[bloodline]

Hunk BSS doesn't really do much beyond inidicating a new hunk to follow, though it must be present to keep the hunk count in sync.

In principle, HUNK BSS could also take reloc information, though its use might be a bit dubious. There are also 16-bit relocation information hunks, though most linkers do not generate them.

My seg loader, flags up if it finds a hunk I don't currently support... so it is easy enough just to add support for other hunk types as I find them.

It depends, and you cannot count on this. It is rather the matter of the compiler switches that were used to compile the command.

I wonder a bit why you want to reinvent the wheel. Vamos exists already.

Several reasons;
Firstly, I am doing this for fun! it's quite a complex project and very satisfying.
Secondly, I have no time or motivation to learn Python, figure out how Chris has structured Vamos and then figure out how to make his ideas fit with mine... like integrating Vamos with my ADFBench would be particularly uninteresting for me.
Thirdly, Vamos is essentially abandoned.

This has been a lot of fun, I'm particularly pleased with my exec.library -> Obj-C method calling interface

You will soon notice that it doesn't stop there. To support all programs, you will need to provide task switching, which is not quite that easy and requires a good abstraction in the emulator part. "Vamos" doesn't do that, and more or less "sneaks around it" by implementing one of the dos.library functions that actually depend on it (namely, SystemTagList()) in a different (non-conforming) way, and others (namely CreateProc()) not at all.

You are quite right, Vamos doesn't really work how I want it to. Though Full credit to his jumptable idea, it's the perfect solution for me (my original idea was to trap the CPU when it tried to read from an address in the jumptable area, which is a terrible idea).

It was good enough for my use case, even though I had to "help" a couple of binares here and there, for example by providing an alternative command line front end for the Lattice compiler, which uses CreateProc() to start the compiler, optimizer and code generator.

[bloodline]

Actually, pyhton is a cute and even quite powerful language and certainly worth learning. It's not wasted time. At times, vamos is really overenginered, though.

I know a lot of people who love Python, but I find it a great deal of effort to read... which after a day at work (I'm a business analyst) is not as much fun to read as C/C++/Obj-C.

Huh, is it? Not by me, at least. I wouldn't say that I'm continuing the project on my own, but at this time, quite a bit already depends on it, and I keep fixing the problems I identified.

apologies, I didn't realise you were contributing to the project! I was going just by the Lallafa blog. I, obviously, think it is an awesome project, but I feel the end goals are quite different.

Vamos uses the line-A traps as entry points from the emulated 68K CPU to the "real world" python code. This is as such a good idea, but you get twists in your brain as soon as you need to do the reverse: Call 68K code from python code. Unfortunately, it is necessary in a couple of places, namely for implementing functions like RawDoFmt().

The current mechanism for that is quite a stunt and implies patching the 68K return stack to redirect the 68K where you want it for executing code.

I haven't really thought about the reverse, that will be fun to figure out

The hardest part for me is the AmigaDos stuff with I never really did anything with when I used to write on the Amiga!