Looking for Extraction tools for 7z and Shrinkler

[Brian]

It pipes compressed input to decompressed output, so you need to run it thusly:
xzminidec ram:file

Thanks, program worked fine but it sadly also uses too much memory when extract the stored archive for this application.

[Brian]

Really?  It should use hardly anything: http://tukaani.org/xz/embedded.html
Maybe you're using too big a dictionary, as that needs to fit in memory too?

Yes really, as said I want it to work on a 1MB machine... in a bad scenario that mean about 670kb left after booting the floppy and it ran out of memory on me.

Don't know what you mean about directory... I used the -1 option for xz as it was the only one I could see impacted memory on extraction.

[Brian]

7zip also have command line so can set dictionary to 4K if I want however even at 64K I lose all the gains and then some compared to just using 7z in a single sweep w/o first storing it.

[Brian]

Hmmm... might be an idea to try combining compression methods.

If you stored the overall delivey folder arrangement as an LHA full of empty folders... and then did a series of 7-zips mini archives each with a folder content... then each folder gets its own mini dictionary, just for dealing with system files of that particular file type.

It should cut down on your system resource requirements. You may find two different compressors deal with each delivered system folder competetively.

Even if you stick with just 7 zip, that might be a way forward? What you are attempting is 6:1 compression. Better than that really, because you have to include decempression tools on your one delivering disk, as well as the 6 floppies worth of data.

Ya 6:1 compression ratio is never going to happen... I get about 2.5:1 atm. With a bit of clever optimization that gives about 2.4MB on the HDD after installation. What you won't get is A2322 support, HDBackup/BRU, Printers (except Generic), _bullet Fonts and Dan/Esp/Ita/Ned/Nor/Por Locals.

I've split the data into 4 archives, a small boot archive (everything needed to boot and use the contents on the disk), gfx (icons and fonts), txt (Dosdrivers, locales, scripts etc.) and bin (rest). By splitting so similar type of data is together I got a bit better compression anything else I've tried when it comes to store then compress either don't give better compression ratio or cost too much memory to extract.

[Brian]

The advantage of XZ is it's the same compression but not using as much memory to decompress, and XZ compressing a "stored" archive is more efficient then 7-Zipping the files (as there's more data to work with).

You should be able to set the dictionary to 256K with XZ and get better compression whilst still being able to extract.

(btw, when I said compress using 7-Zip I actually meant compress using the 7-Zip GUI tool, but using XZ compression... although I realise it doesn't read like that!)

I understood that after a while... XZ refusing to accept the file was a hint... bit I was working with a 6 y/o 7zip that didn't have xz implemented. Good now that's updated.

Another optimisation would be to use a BCJ filter.  However, there isn't one for 68k so it would need writing... but that should help the compression a fair bit with the executables.

I'm all for better compression with little to no extra memory usage. Tell me when you find a BCJ filter.

[Brian]

The advantage of XZ is it's the same compression but not using as much memory to decompress, and XZ compressing a "stored" archive is more efficient then 7-Zipping the files (as there's more data to work with).

You should be able to set the dictionary to 256K with XZ and get better compression whilst still being able to extract.

(btw, when I said compress using 7-Zip I actually meant compress using the 7-Zip GUI tool, but using XZ compression... although I realise it doesn't read like that!)

Ok I stand corrected, I was able to gain 17KB (after deducting the the 13KB for shrinkler crunched xzminidec) using store then compress without running out of memory. It does mean I have to up the requirement of HDD space from 3MB to 5MB but that's cool (even back in the days Commodore thought an OS partition should be at least 8MB).

And I'll probably look at a different tool to store the files than 7z now that the compression is taken care of by xz since there's smaller unpackers than 7zdec.

[Brian]

Should be easy enough to write for somebody with knowledge of 68k asm.
PPC version: http://git.tukaani.org/?p=xz.git;a=blob;f=src/liblzma/simple/powerpc.c;h=54dfbf102878ef53ecf694ae3eb7e68473811be1;hb=HEAD

All it does is replace relative offsets in Branch/Call/Jump instructions with absolute values.  The theory being that subroutines are referenced multiple times, and this makes the code more repetitive and compressible.  The filter than undoes this change when it is decompressed.

If you're doing a store then an Amiga-specific archiver is best as it will keep all the special attributes.  LhA would be my suggestion, maybe even go back and find a really old version as it'll be smaller.  The newer ones have all sorts of extra compression schemes and things that won't be relevant.

We can always hope someone makes that filter, I can't do it myselv and I'm not holding my breath.

Think I'll go with LZX as I had before... with UnLZX being only 10KB Shrinkler compressed it's hard to beat.

[Brian]

I might be able to work it out given enough time but I also might screw it up spectacularly.  Hence why ideally it needs somebody who knows about 68k asm.

In the meantime I've shaved 5K off xzminidec for you. (by recompiling optimising for size and then stripping the binary, hopefully this hasn't broken anything!)

I'm sure you'd screw it up far less than I would.

Thanks for the slimmer zxminidec. It translated into 624b saving when shrinkler had run through it but every block counts so this helps... if it's not broken.