Looking for Extraction tools for 7z and Shrinkler

[chris]

Untested: http://www.cy2.uk/tmp/7zdec_os3.lha

[chris]

I'd love to test it but the archive is corrupt so can't extract the file, any chance you can upload it again?

Ah, it probably isn't corrupt - For quickness, I compressed it using a Linux LhA which uses a compression method the Amiga command-line LhA doesn't support.  You should be able to extract it with UnArc.  Otherwise, I'll re-compress and re-upload it this evening.

It seem to be a bit on the large side for an extraction tool (and double the size of the PPC version) but if it's not dependent on some external library it might still help me save some space and every KB counts... fingers crossed it'll be double digit which would be just awesome.

That's about as small as it will get, I might be able to toggle some optimisations but it's unlikely to make much difference.  It's the decompression tool from the LZMA SDK, ie. the bare minimum.  It's less than 80K.

edit I suppose you could PowerPacker the 7zDec executable

[chris]

None of the 7z tools on aminet  can extract large archives. Tried both OS3 and OS4 ones. And also the XAD one

Getting off-topic, but I'd be interested to know what you can't extract with the XAD 7-Zip module. It does need enough RAM (inc. VMEM) to extract files and probably hold the original archive too, so anything over about 2GB won't extract (also I doubt XAD will work with anything over 4GB). Is that the sort of size you're talking about?

[chris]

I will check and get back to you, but I think it was a whdloadpack starting with a letter I do not remember.

OK thanks.

[chris]

Tried the file responsible. Filesize is 332,299,367. It is called WHDLoad Games Update - 2011-07-18.7z.  UnARC gives a  not enough mem requester when I try to extract the archive or a file from it to my hdd. I have 377,922,544 free fastmem available. I will try with some other 7z files later.

I can't find that file anywhere so am unable to try it myself.  However, given the sizes most of your memory will be taken up with the original archive and overhead.  I'd expect it to extract small files from it no problem.  If you try under OS4 with a swap partition then it should extract the whole thing.

Without being able to try it I can't tell if there's some other problem with that file.  Ideally I need to get the plugin to be able to read and write direct to/from disk in order to reduce memory usage, but that's not likely to happen any time soon.

[chris]

I wish it did not need this much memory. Can't it extract files in chunks that would fit inside available memory?

edit actually, no, it doesn't work as I was thinking it did.
The extraction function here would need to be rewritten so it doesn't allocate all the memory it needs for the entire decoded file, and instead decompress in chunks. That's a modification I don't have the time or energy for.

[chris]

I did get quite the saving when storing all before archiving it in a second pass with 7z however 7zdec req. too much memory to extract that so guess it's not just the "blocksize" but rather what fits in that block that is the key to memoryusage with 7zdec's routines.

This is where xzminidec is useful.

[chris]

I could not get it to run, just hangs after you run it with no output, same when trying with ? after or a filename after. Tried to increase the stack to 100000 but it didn't change the behavior.

It pipes compressed input to decompressed output, so you need to run it thusly:
xzminidec ram:file

[chris]

Thanks, program worked fine but it sadly also uses too much memory when extract the stored archive for this application.

Really?  It should use hardly anything: http://tukaani.org/xz/embedded.html
Maybe you're using too big a dictionary, as that needs to fit in memory too?

[chris]

Yes really, as said I want it to work on a 1MB machine... in a bad scenario that mean about 670kb left after booting the floppy and it ran out of memory on me.

Don't know what you mean about directory... I used the -1 option for xz as it was the only one I could see impacted memory on extraction.

Hmm...

Code: [Select]

Because the settings may vary, the memory usage may vary too. The following
    table lists the maximum memory usage of each preset level, which won't be exceeded even in future versions of xz.
FIXME: The table below is just a rough idea.

            Preset Compression Decompression
            -0 6 MiB 1 MiB
            -1 6 MiB 1 MiB
            -2 10 MiB 1 MiB
            -3 20 MiB 2 MiB
            -4 30 MiB 3 MiB
            -5 60 MiB 6 MiB
            -6 100 MiB 10 MiB
            -7 200 MiB 20 MiB
            -8 400 MiB 40 MiB
            -9 800 MiB 80 MiB
https://linux.die.net/man/1/xz

What I think you need to specify is --lzma2=dict=64K (or some other smallish size)

If you compress using 7-Zip you get to see the dictionary size on the GUI (but it doesn't give you any options below 1MB except 64K, whereas the command line tool will accept anything above 4K, which gives a lot more flexibility)

[chris]

7zip also have command line so can set dictionary to 4K if I want however even at 64K I lose all the gains and then some compared to just using 7z in a single sweep w/o first storing it.

The advantage of XZ is it's the same compression but not using as much memory to decompress, and XZ compressing a "stored" archive is more efficient then 7-Zipping the files (as there's more data to work with).

You should be able to set the dictionary to 256K with XZ and get better compression whilst still being able to extract.

(btw, when I said compress using 7-Zip I actually meant compress using the 7-Zip GUI tool, but using XZ compression... although I realise it doesn't read like that!)

[chris]

It should cut down on your system resource requirements. You may find two different compressors deal with each delivered system folder competetively.

Another optimisation would be to use a BCJ filter.  However, there isn't one for 68k so it would need writing... but that should help the compression a fair bit with the executables.

[chris]

I'm all for better compression with little to no extra memory usage. Tell me when you find a BCJ filter.

Should be easy enough to write for somebody with knowledge of 68k asm.
PPC version: http://git.tukaani.org/?p=xz.git;a=blob;f=src/liblzma/simple/powerpc.c;h=54dfbf102878ef53ecf694ae3eb7e68473811be1;hb=HEAD

All it does is replace relative offsets in Branch/Call/Jump instructions with absolute values.  The theory being that subroutines are referenced multiple times, and this makes the code more repetitive and compressible.  The filter than undoes this change when it is decompressed.

And I'll probably look at a different tool to store the files than 7z now that the compression is taken care of by xz since there's smaller unpackers than 7zdec.

If you're doing a store then an Amiga-specific archiver is best as it will keep all the special attributes.  LhA would be my suggestion, maybe even go back and find a really old version as it'll be smaller.  The newer ones have all sorts of extra compression schemes and things that won't be relevant.

[chris]

We can always hope someone makes that filter, I can't do it myselv and I'm not holding my breath.

I might be able to work it out given enough time but I also might screw it up spectacularly.  Hence why ideally it needs somebody who knows about 68k asm.

In the meantime I've shaved 5K off xzminidec for you. (by recompiling optimising for size and then stripping the binary, hopefully this hasn't broken anything!)