Looking for Extraction tools for 7z and Shrinkler

[nyteschayde]

I recently compiled rzip and uploaded it to Aminet. With a little effort, I could probably get it to compile without ixemul.library as well. I have found that using lha in place of tar (simply because tar is not common on the Amiga) and storing, not compressing, the files, followed by rziping the resulting archive produces massive savings.

For the ADE repack I uploaded, previously compressed with zip, the archive went from 20MB to 13MB using this technique. Something to consider.

[Brian]

I recently compiled rzip and uploaded it to Aminet. With a little effort, I could probably get it to compile without ixemul.library as well. I have found that using lha in place of tar (simply because tar is not common on the Amiga) and storing, not compressing, the files, followed by rziping the resulting archive produces massive savings.

For the ADE repack I uploaded, previously compressed with zip, the archive went from 20MB to 13MB using this technique. Something to consider.

I'd love to have a look at it if you compile it to be library/fpu independent and 68000 compatible.

I did get quite the saving when storing all before archiving it in a second pass with 7z however 7zdec req. too much memory to extract that so guess it's not just the "blocksize" but rather what fits in that block that is the key to memoryusage with 7zdec's routines.

[nyteschayde]

I'd love to have a look at it if you compile it to be library/fpu independent and 68000 compatible.

I did get quite the saving when storing all before archiving it in a second pass with 7z however 7zdec req. too much memory to extract that so guess it's not just the "blocksize" but rather what fits in that block that is the key to memoryusage with 7zdec's routines.

I may have time this weekend to make a pass. I'll see what I can do.

[chris]

I did get quite the saving when storing all before archiving it in a second pass with 7z however 7zdec req. too much memory to extract that so guess it's not just the "blocksize" but rather what fits in that block that is the key to memoryusage with 7zdec's routines.

This is where xzminidec is useful.

[Brian]

I may have time this weekend to make a pass. I'll see what I can do.

I'd appreciate it.

This is where xzminidec is useful.

I could not get it to run, just hangs after you run it with no output, same when trying with ? after or a filename after. Tried to increase the stack to 100000 but it didn't change the behavior.

[chris]

I could not get it to run, just hangs after you run it with no output, same when trying with ? after or a filename after. Tried to increase the stack to 100000 but it didn't change the behavior.

It pipes compressed input to decompressed output, so you need to run it thusly:
xzminidec ram:file

[Brian]

It pipes compressed input to decompressed output, so you need to run it thusly:
xzminidec ram:file

Thanks, program worked fine but it sadly also uses too much memory when extract the stored archive for this application.

[chris]

Thanks, program worked fine but it sadly also uses too much memory when extract the stored archive for this application.

Really?  It should use hardly anything: http://tukaani.org/xz/embedded.html
Maybe you're using too big a dictionary, as that needs to fit in memory too?

[Brian]

Really?  It should use hardly anything: http://tukaani.org/xz/embedded.html
Maybe you're using too big a dictionary, as that needs to fit in memory too?

Yes really, as said I want it to work on a 1MB machine... in a bad scenario that mean about 670kb left after booting the floppy and it ran out of memory on me.

Don't know what you mean about directory... I used the -1 option for xz as it was the only one I could see impacted memory on extraction.

[chris]

Yes really, as said I want it to work on a 1MB machine... in a bad scenario that mean about 670kb left after booting the floppy and it ran out of memory on me.

Don't know what you mean about directory... I used the -1 option for xz as it was the only one I could see impacted memory on extraction.

Hmm...

Code: [Select]

Because the settings may vary, the memory usage may vary too. The following
    table lists the maximum memory usage of each preset level, which won't be exceeded even in future versions of xz.
FIXME: The table below is just a rough idea.

            Preset Compression Decompression
            -0 6 MiB 1 MiB
            -1 6 MiB 1 MiB
            -2 10 MiB 1 MiB
            -3 20 MiB 2 MiB
            -4 30 MiB 3 MiB
            -5 60 MiB 6 MiB
            -6 100 MiB 10 MiB
            -7 200 MiB 20 MiB
            -8 400 MiB 40 MiB
            -9 800 MiB 80 MiB
https://linux.die.net/man/1/xz

What I think you need to specify is --lzma2=dict=64K (or some other smallish size)

If you compress using 7-Zip you get to see the dictionary size on the GUI (but it doesn't give you any options below 1MB except 64K, whereas the command line tool will accept anything above 4K, which gives a lot more flexibility)

[Brian]

7zip also have command line so can set dictionary to 4K if I want however even at 64K I lose all the gains and then some compared to just using 7z in a single sweep w/o first storing it.

[Pat the Cat]

Hmmm... might be an idea to try combining compression methods.

If you stored the overall delivey folder arrangement as an LHA full of empty folders... and then did a series of 7-zips mini archives each with a folder content... then each folder gets its own mini dictionary, just for dealing with system files of that particular file type.

It should cut down on your system resource requirements. You may find two different compressors deal with each delivered system folder competetively.

Even if you stick with just 7 zip, that might be a way forward? What you are attempting is 6:1 compression. Better than that really, because you have to include decempression tools on your one delivering disk, as well as the 6 floppies worth of data.

[chris]

7zip also have command line so can set dictionary to 4K if I want however even at 64K I lose all the gains and then some compared to just using 7z in a single sweep w/o first storing it.

The advantage of XZ is it's the same compression but not using as much memory to decompress, and XZ compressing a "stored" archive is more efficient then 7-Zipping the files (as there's more data to work with).

You should be able to set the dictionary to 256K with XZ and get better compression whilst still being able to extract.

(btw, when I said compress using 7-Zip I actually meant compress using the 7-Zip GUI tool, but using XZ compression... although I realise it doesn't read like that!)

[chris]

It should cut down on your system resource requirements. You may find two different compressors deal with each delivered system folder competetively.

Another optimisation would be to use a BCJ filter.  However, there isn't one for 68k so it would need writing... but that should help the compression a fair bit with the executables.

[Brian]

Hmmm... might be an idea to try combining compression methods.

If you stored the overall delivey folder arrangement as an LHA full of empty folders... and then did a series of 7-zips mini archives each with a folder content... then each folder gets its own mini dictionary, just for dealing with system files of that particular file type.

It should cut down on your system resource requirements. You may find two different compressors deal with each delivered system folder competetively.

Even if you stick with just 7 zip, that might be a way forward? What you are attempting is 6:1 compression. Better than that really, because you have to include decempression tools on your one delivering disk, as well as the 6 floppies worth of data.

Ya 6:1 compression ratio is never going to happen... I get about 2.5:1 atm. With a bit of clever optimization that gives about 2.4MB on the HDD after installation. What you won't get is A2322 support, HDBackup/BRU, Printers (except Generic), _bullet Fonts and Dan/Esp/Ita/Ned/Nor/Por Locals.

I've split the data into 4 archives, a small boot archive (everything needed to boot and use the contents on the disk), gfx (icons and fonts), txt (Dosdrivers, locales, scripts etc.) and bin (rest). By splitting so similar type of data is together I got a bit better compression anything else I've tried when it comes to store then compress either don't give better compression ratio or cost too much memory to extract.

[Brian]

The advantage of XZ is it's the same compression but not using as much memory to decompress, and XZ compressing a "stored" archive is more efficient then 7-Zipping the files (as there's more data to work with).

You should be able to set the dictionary to 256K with XZ and get better compression whilst still being able to extract.

(btw, when I said compress using 7-Zip I actually meant compress using the 7-Zip GUI tool, but using XZ compression... although I realise it doesn't read like that!)

I understood that after a while... XZ refusing to accept the file was a hint... bit I was working with a 6 y/o 7zip that didn't have xz implemented. Good now that's updated.

Another optimisation would be to use a BCJ filter.  However, there isn't one for 68k so it would need writing... but that should help the compression a fair bit with the executables.

I'm all for better compression with little to no extra memory usage. Tell me when you find a BCJ filter.