Amiga.org
Amiga computer related discussion => Amiga Software Issues and Discussion => Topic started by: Brian on January 23, 2017, 07:18:15 PM
-
I'm using LZX to pack different kind of data onto a floppy and use UnLZX (also need to fit the disk) to unpack the data. I know 7z can give me roughly 60-100KB better compression, something I'd gladly take since I have more I'd love to cram onto the thing, so I wonder if there exist a tiny standalone un7z tool? Checked Aminet but the programs I find there usually req. a 100+KB library to work and ultimately result in a bigger package (archive+extractiontool+libs).
On a sidenote, does there exist any small decruncher for Shrinkler compressed programs?
Both need to work with 68000 without FPU.
-
Untested: http://www.cy2.uk/tmp/7zdec_os3.lha
-
I'd love to test it but the archive is corrupt so can't extract the file, any chance you can upload it again?
It seem to be a bit on the large side for an extraction tool (and double the size of the PPC version) but if it's not dependent on some external library it might still help me save some space and every KB counts... fingers crossed it'll be double digit which would be just awesome. :D
-
On a sidenote, does there exist any small decruncher for Shrinkler compressed programs?
Doesn't Shrinkler itself work for this purpose? And the archive comes with source code for the decruncher, so you could basically also make your own tool.
-
Doesn't Shrinkler itself work for this purpose? And the archive comes with source code for the decruncher, so you could basically also make your own tool.
The thing is I'd like to revert back to the original executable file from a Shrinkler crunched file. You'r right there is a source in the archive but since I can't code it doesn't really help me.
-
I'd love to test it but the archive is corrupt so can't extract the file, any chance you can upload it again?
Ah, it probably isn't corrupt - For quickness, I compressed it using a Linux LhA which uses a compression method the Amiga command-line LhA doesn't support. You should be able to extract it with UnArc. Otherwise, I'll re-compress and re-upload it this evening.
It seem to be a bit on the large side for an extraction tool (and double the size of the PPC version) but if it's not dependent on some external library it might still help me save some space and every KB counts... fingers crossed it'll be double digit which would be just awesome. :D
That's about as small as it will get, I might be able to toggle some optimisations but it's unlikely to make much difference. It's the decompression tool from the LZMA SDK, ie. the bare minimum. It's less than 80K.
edit I suppose you could PowerPacker the 7zDec executable :)
-
I'm using LZX to pack different kind of data onto a floppy and use UnLZX (also need to fit the disk) to unpack the data. .
On a sidenote, does there exist any small decruncher for Shrinkler compressed programs?
---
Both need to work with 68000 without FPU.
Here's an oddity - different compressor programs vary in efficiency on different data types.
I reckon, 7z, even with a library, will do a better job. dear old Lentle-Ziff-Huffman isn't what it used to be, and that's what most Amiga crunchers are based around.
If you setup all your files at once, so they end up as one big file, that will cut a lot of redundancy out, for files of a similar type. Especially if they are not too big to begin with.
One datatype that will NEVER crunch is raw fax code. It's already 2-1 compressed or better to begin with.
And finally, do make it clear to the end user that they have to decrunch the stuff TO somewhere. A bit of text to explain how to use, in a readme.txt file. That's worth it. It will take up a couple of disk sectors (which is a good way to think about using floppy disks efficiently) but don't rely on the person with the disk to have a clue what to do with it.
I hope one day somebody codes up IFS compression, but that's not going to happen in a hurry. It's not exactly ideal for lossless, which is what you need for executables. It's a bit fuzzy.
-
Here's an oddity - different compressor programs vary in efficiency on different data types.
I reckon, 7z, even with a library, will do a better job. dear old Lentle-Ziff-Huffman isn't what it used to be, and that's what most Amiga crunchers are based around.
If you setup all your files at once, so they end up as one big file, that will cut a lot of redundancy out, for files of a similar type. Especially if they are not too big to begin with.
One datatype that will NEVER crunch is raw fax code. It's already 2-1 compressed or better to begin with.
And finally, do make it clear to the end user that they have to decrunch the stuff TO somewhere. A bit of text to explain how to use, in a readme.txt file. That's worth it. It will take up a couple of disk sectors (which is a good way to think about using floppy disks efficiently) but don't rely on the person with the disk to have a clue what to do with it.
I hope one day somebody codes up IFS compression, but that's not going to happen in a hurry. It's not exactly ideal for lossless, which is what you need for executables. It's a bit fuzzy.
Before when I checked I get about 60K better compression with 7z than LZX but last time I ran it I got about 100K so don't know why last time was any better... however most unpackers rely on 130+KB libraries plus the unachiver so thus far I've not found something that rivals LZX when it comes to data on a single floppy.
In short I try to put all six OS3.1 floppys onto a single DD floppy and have that work as a standalone install disk that you can use first time setting up the HDD to install. I want it to work on 68000 and requier no more than 1MB memory (atm I uses less than 768KB). I've optimized the heck out of it and I'm at where there is nothing left I can do that would save space (that I'm willing to do). Except if I can get the already compressed files on the floppy to take even less space. Replace LZX archives with z7 and replace Implode crunched files with Shrinkler. There's space to be saved but it also depend on the size of the unpacking tools (Deplode added only 900B to the LZX archive it's in) and UnLZX is crunched to only 10KB and need no libraries... something I've not seen anything rival until Chris's post.
-
If you still use Imploader just remember the 040/060 problem:
http://aminet.net/package/util/libs/explode-7 (http://aminet.net/package/util/libs/explode-7)
-
If you still use Imploader just remember the 040/060 problem:
http://aminet.net/package/util/libs/explode-7 (http://aminet.net/package/util/libs/explode-7)
The Imploded crunched files have internal lib so no external lib required and that doesn't seem to have issues with 040/060.
-
In short I try to put all six OS3.1 floppys onto a single DD floppy and have that work as a standalone install disk that you can use first time setting up the HDD to install. I want it to work on 68000 and requier no more than 1MB memory (atm I uses less than 768KB). I've optimized the heck out of it and I'm at where there is nothing left I can do that would save space (that I'm willing to do). Except if I can get the already compressed files on the floppy to take even less space. Replace LZX archives with z7 and replace Implode crunched files with Shrinkler. There's space to be saved but it also depend on the size of the unpacking tools (Deplode added only 900B to the LZX archive it's in) and UnLZX is crunched to only 10KB and need no libraries... something I've not seen anything rival until Chris's post.
Without commenting on the lawfulness of the chosen data... Which would be perfectly OK if the person using it has an actual ROM chip somewhere in their collection. (Will Amiga Inc ever sue Mac over that, I wonder? Seems fair, it was the other way around in the past). Anyway...
That is your problem, doing single fles rather than a single joined up one. When you use any data sector, any unused space is wasted on the last sector. That is why arranging the data into a single set, then compressing that set into a single file, is going to save you space.
The snags are two fold - first, you need a fast Amiga with lots of fast RAM to set it up - second, your compressor has to first uncompress the archive, the correctly chop it up into single files and copy them to the HDD. You have set no margin for error on doing that on a 1MB single floppy with a HDD attached. But, I admire your thoroughness in pursuing that goal - trying to get it universal.
Here's a simpler way - do it as 2 disks. One as the uncruncher. The next one, when booted, splits up the decompressed file, makes a new set of sub directories, and copies all the split files into the right places. That might just work, and give the end user a nice front end when setting the thing up. The snag is, the decrunching has to happen on one file, which has to be gradually read in, decompressed, and copied to the hard drive. Very demanding.
It does mean two boots from different floppies, but it gives both you and the end user some control over how their hard drives will be setup. That might not seem an issue. Well, it is if the end user is running a real 1.3 Amiga. Having a warning saying "this isn't going to work without a ROM change" would be very helpful to noobs with Amigas, for instance.
Even if you don't for the single file option (it is very challenging), a second floppy for excess data and setup might be required anyway. For files crunches that won't fit, and also for the setup. It would save one reboot, so might be a bit quicker for the end user.
-
Is it possible, and if so have you tried to remove files that you won't need? (e.g. any unused locales, screen modes, etc)?
-
The Imploded crunched files have internal lib so no external lib required and that doesn't seem to have issues with 040/060.
Ah okay, it's been a while since I used it (1990's)! I'll have to play around with it again at some point. So there must be a choice then between external library (explode) and internal decompressor (inside executable)... and if you have more than one crunched executable then the library would be more space efficient.
So anyway, you're trying to squeeze the OS3.1 disk set onto just one floppy? It certainly sounds ambitious. How small have you gotten it up to yet?
-
Is it possible, and if so have you tried to remove files that you won't need? (e.g. any unused locales, screen modes, etc)?
Ow I have to do that or it wouldn't fit for sure. Almost all Locals and printers as well as a good chunk of fonts and a few other things. Removed duplicate mountlist and icons to copy/create and reposition them later. Created a custom install script. And crunched/packed just about everything left.
Ah okay, it's been a while since I used it (1990's)! I'll have to play around with it again at some point. So there must be a choice then between external library (explode) and internal decompressor (inside executable)... and if you have more than one crunched executable then the library would be more space efficient.
So anyway, you're trying to squeeze the OS3.1 disk set onto just one floppy? It certainly sounds ambitious. How small have you gotten it up to yet?
I only have HDToolbox Imploder crunched since I don't want it to be crunched on the HDD and I can use deploder for that and having the imploder lib in the exe doesn't add much to the filesize so more spacesaving that way. UnLZX is Shrinkler compressed and doesn't need any library. Rest is either in LZX archive or can't be crunched/packed due to the nature of the file (such as startup-seq, a readme file, L:fastfilesystem)
It fits right now but as said to puppypc I've had to remove some nonessential files, but I want that at a minimum so any space I can save goes towards making it more complete again.
Without commenting on the lawfulness of the chosen data... Which would be perfectly OK if the person using it has an actual ROM chip somewhere in their collection. (Will Amiga Inc ever sue Mac over that, I wonder? Seems fair, it was the other way around in the past). Anyway...
That is your problem, doing single fles rather than a single joined up one. When you use any data sector, any unused space is wasted on the last sector. That is why arranging the data into a single set, then compressing that set into a single file, is going to save you space.
The snags are two fold - first, you need a fast Amiga with lots of fast RAM to set it up - second, your compressor has to first uncompress the archive, the correctly chop it up into single files and copy them to the HDD. You have set no margin for error on doing that on a 1MB single floppy with a HDD attached. But, I admire your thoroughness in pursuing that goal - trying to get it universal.
Here's a simpler way - do it as 2 disks. One as the uncruncher. The next one, when booted, splits up the decompressed file, makes a new set of sub directories, and copies all the split files into the right places. That might just work, and give the end user a nice front end when setting the thing up. The snag is, the decrunching has to happen on one file, which has to be gradually read in, decompressed, and copied to the hard drive. Very demanding.
It does mean two boots from different floppies, but it gives both you and the end user some control over how their hard drives will be setup. That might not seem an issue. Well, it is if the end user is running a real 1.3 Amiga. Having a warning saying "this isn't going to work without a ROM change" would be very helpful to noobs with Amigas, for instance.
Even if you don't for the single file option (it is very challenging), a second floppy for excess data and setup might be required anyway. For files crunches that won't fit, and also for the setup. It would save one reboot, so might be a bit quicker for the end user.
About legal... as long as I have it for personal use and for friends and others who own the WB3.1 I see no legal problems. Sure it might eventually leak onto the internet but that won't be my doing.
I use LZX merge option and it compresses it nicely... there's actually only about 1.2KB to save from putting everything into one uncompressed archive before compressing it and I decided that the extra memory requirement and decompression time wasn't worth it. That said 7z would compress it way harder so I'm still hopeful it'll be the next space saving step.
Everything fits way too easy onto a HD floppy (very first incarnation of this project) or 2 floppys... I've created a second Extras disk just to make the installation complete again and that one also includes big hdd support, pfs, sfs, , 060 libs, pcmcia cf support, magicwb and a few other addons. But it's optional "extras"... I want the challenge of making it a standalone 1 disk option mainly cause one of the reasons for this project was to remove the need for swapping floppies and another reason was so you don't have a corrupt disk X of 6 preventing you from completing the installation.
-
7zdec helped me save a bit more space by allowing me to use 7zip for the archives instead of LZX. Big thanks to chris!
-
None of the 7z tools on aminet can extract large archives. Tried both OS3 and OS4 ones. And also the XAD one
-
None of the 7z tools on aminet can extract large archives. Tried both OS3 and OS4 ones. And also the XAD one
My archive(s) need to fit a single DD floppy so I wouldn't consider that large, the problem I had was that the tools I found where simply so large with external library that any space saved by the better compression of 7z over LZX was lost.
Chris to the rescue with a port of 7zdec to OS3 with no need for external library and working with 68000. It fit my need very well although it being a bit memory hungry, but I was able to work around that and shave ~63Kb of the initial LZX archive size and with 7zdec crunched to 33Kb (over UnLZX's 10K) there was still about 40K space saved over all.
Now if only I could get hold of a small Shrinkler exe decompress tool I'd most likely be able to save another 12Kb or so.
-
I only have HDToolbox Imploder crunched since I don't want it to be crunched on the HDD and I can use deploder for that and having the imploder lib in the exe doesn't add much to the filesize so more spacesaving that way. UnLZX is Shrinkler compressed and doesn't need any library. Rest is either in LZX archive or can't be crunched/packed due to the nature of the file (such as startup-seq, a readme file, L:fastfilesystem)
FastFileSystem is in ROM too, so that's another file you can safely delete. Granted, it's a minor update to the v39 version in the 3.0 ROM, but I don't know what the benefit would be except minor bug fixes. You can use OS3.1 with a 3.0 ROM, which would explain why it is included on disk as the user could install the updated v40 version to their RDB if they so desire. So on a 3.1 ROM system, the updated v40.1 FastFileSystem isn't useful as it's already in ROM.
-
FastFileSystem is in ROM too, so that's another file you can safely delete. Granted, it's a minor update to the v39 version in the 3.0 ROM, but I don't know what the benefit would be except minor bug fixes. You can use OS3.1 with a 3.0 ROM, which would explain why it is included on disk as the user could install the updated v40 version to their RDB if they so desire. So on a 3.1 ROM system, the updated v40.1 FastFileSystem isn't useful as it's already in ROM.
I wonder if there's not some old A500 HDD controllers software that require FFS on file when setting up the HDD even if it is in ROM, granted it's not the main stream scenario for an OS3.1 installation but still. I've however though about adding an option to decrunch it separately in case it's needed early for HDD setup.
-
I wonder if there's not some old A500 HDD controllers software that require FFS on file when setting up the HDD even if it is in ROM, granted it's not the main stream scenario for an OS3.1 installation but still. I've however though about adding an option to decrunch it separately in case it's needed early for HDD setup.
There are some very old controllers which aren't RDB compatible, but I believe those ones would not have drivers in their ROM either, but I'm not sure. Better to cater for the masses being as though you only have one floppy disk to play with.
It sounds an interesting project this, the kind of thing I'd like to muck around with. :)
-
None of the 7z tools on aminet can extract large archives. Tried both OS3 and OS4 ones. And also the XAD one
Getting off-topic, but I'd be interested to know what you can't extract with the XAD 7-Zip module. It does need enough RAM (inc. VMEM) to extract files and probably hold the original archive too, so anything over about 2GB won't extract (also I doubt XAD will work with anything over 4GB). Is that the sort of size you're talking about?
-
There are some very old controllers which aren't RDB compatible, but I believe those ones would not have drivers in their ROM either, but I'm not sure. Better to cater for the masses being as though you only have one floppy disk to play with.
It sounds an interesting project this, the kind of thing I'd like to muck around with. :)
It's fun but frustrating at times...
What I've had to drop is basically HDBackup/BRU, all printers except for Generic, all localization except for Swedish, Courier, Helvetica, Letter Gothic and _Bullet Fonts and A2232 Serial card support... but I'm still working on shrinking this list even further. ;)
-
I will check and get back to you, but I think it was a whdloadpack starting with a letter I do not remember.
-
I will check and get back to you, but I think it was a whdloadpack starting with a letter I do not remember.
OK thanks.
-
OK thanks.
Tried the file responsible. Filesize is 332,299,367. It is called WHDLoad Games Update - 2011-07-18.7z. UnARC gives a not enough mem requester when I try to extract the archive or a file from it to my hdd. I have 377,922,544 free fastmem available. I will try with some other 7z files later.
-
Tried the file responsible. Filesize is 332,299,367. It is called WHDLoad Games Update - 2011-07-18.7z. UnARC gives a not enough mem requester when I try to extract the archive or a file from it to my hdd. I have 377,922,544 free fastmem available. I will try with some other 7z files later.
I can't find that file anywhere so am unable to try it myself. However, given the sizes most of your memory will be taken up with the original archive and overhead. I'd expect it to extract small files from it no problem. If you try under OS4 with a swap partition then it should extract the whole thing.
Without being able to try it I can't tell if there's some other problem with that file. Ideally I need to get the plugin to be able to read and write direct to/from disk in order to reduce memory usage, but that's not likely to happen any time soon.
-
I wish it did not need this much memory. Can't it extract files in chunks that would fit inside available memory?
-
I wish it did not need this much memory. Can't it extract files in chunks that would fit inside available memory?
I would love that also but sadly right now it doesn't work like that. However you can limit the amount of memory needed if you limit the blocksize of the archive itself though you lose a fair bit of compression. 7zip paramter s=256000b aught to be pretty safe for most 1MB systems.
-
I wish it did not need this much memory. Can't it extract files in chunks that would fit inside available memory?
edit actually, no, it doesn't work as I was thinking it did.
The extraction function here (https://github.com/chris-y/xad7z/blob/master/C/7zArcIn.c) would need to be rewritten so it doesn't allocate all the memory it needs for the entire decoded file, and instead decompress in chunks. That's a modification I don't have the time or energy for.
-
I recently compiled rzip and uploaded it to Aminet. With a little effort, I could probably get it to compile without ixemul.library as well. I have found that using lha in place of tar (simply because tar is not common on the Amiga) and storing, not compressing, the files, followed by rziping the resulting archive produces massive savings.
For the ADE repack I uploaded, previously compressed with zip, the archive went from 20MB to 13MB using this technique. Something to consider.
-
I recently compiled rzip and uploaded it to Aminet. With a little effort, I could probably get it to compile without ixemul.library as well. I have found that using lha in place of tar (simply because tar is not common on the Amiga) and storing, not compressing, the files, followed by rziping the resulting archive produces massive savings.
For the ADE repack I uploaded, previously compressed with zip, the archive went from 20MB to 13MB using this technique. Something to consider.
I'd love to have a look at it if you compile it to be library/fpu independent and 68000 compatible.
I did get quite the saving when storing all before archiving it in a second pass with 7z however 7zdec req. too much memory to extract that so guess it's not just the "blocksize" but rather what fits in that block that is the key to memoryusage with 7zdec's routines.
-
I'd love to have a look at it if you compile it to be library/fpu independent and 68000 compatible.
I did get quite the saving when storing all before archiving it in a second pass with 7z however 7zdec req. too much memory to extract that so guess it's not just the "blocksize" but rather what fits in that block that is the key to memoryusage with 7zdec's routines.
I may have time this weekend to make a pass. I'll see what I can do.
-
I did get quite the saving when storing all before archiving it in a second pass with 7z however 7zdec req. too much memory to extract that so guess it's not just the "blocksize" but rather what fits in that block that is the key to memoryusage with 7zdec's routines.
This is where xzminidec (http://www.cy2.uk/tmp/xzminidec_os3.lha) is useful.
-
I may have time this weekend to make a pass. I'll see what I can do.
I'd appreciate it.
This is where xzminidec (http://www.cy2.uk/tmp/xzminidec_os3.lha) is useful.
I could not get it to run, just hangs after you run it with no output, same when trying with ? after or a filename after. Tried to increase the stack to 100000 but it didn't change the behavior.
-
I could not get it to run, just hangs after you run it with no output, same when trying with ? after or a filename after. Tried to increase the stack to 100000 but it didn't change the behavior.
It pipes compressed input to decompressed output, so you need to run it thusly:
xzminidec ram:file
-
It pipes compressed input to decompressed output, so you need to run it thusly:
xzminidec ram:file
Thanks, program worked fine but it sadly also uses too much memory when extract the stored archive for this application. :(
-
Thanks, program worked fine but it sadly also uses too much memory when extract the stored archive for this application. :(
Really? It should use hardly anything: http://tukaani.org/xz/embedded.html
Maybe you're using too big a dictionary, as that needs to fit in memory too?
-
Really? It should use hardly anything: http://tukaani.org/xz/embedded.html
Maybe you're using too big a dictionary, as that needs to fit in memory too?
Yes really, as said I want it to work on a 1MB machine... in a bad scenario that mean about 670kb left after booting the floppy and it ran out of memory on me.
Don't know what you mean about directory... I used the -1 option for xz as it was the only one I could see impacted memory on extraction.
-
Yes really, as said I want it to work on a 1MB machine... in a bad scenario that mean about 670kb left after booting the floppy and it ran out of memory on me.
Don't know what you mean about directory... I used the -1 option for xz as it was the only one I could see impacted memory on extraction.
Hmm...
Because the settings may vary, the memory usage may vary too. The following
table lists the maximum memory usage of each preset level, which won't be exceeded even in future versions of xz.
FIXME: The table below is just a rough idea.
Preset Compression Decompression
-0 6 MiB 1 MiB
-1 6 MiB 1 MiB
-2 10 MiB 1 MiB
-3 20 MiB 2 MiB
-4 30 MiB 3 MiB
-5 60 MiB 6 MiB
-6 100 MiB 10 MiB
-7 200 MiB 20 MiB
-8 400 MiB 40 MiB
-9 800 MiB 80 MiB
https://linux.die.net/man/1/xz
What I think you need to specify is --lzma2=dict=64K (or some other smallish size)
If you compress using 7-Zip you get to see the dictionary size on the GUI (but it doesn't give you any options below 1MB except 64K, whereas the command line tool will accept anything above 4K, which gives a lot more flexibility)
-
7zip also have command line so can set dictionary to 4K if I want however even at 64K I lose all the gains and then some compared to just using 7z in a single sweep w/o first storing it.
-
Hmmm... might be an idea to try combining compression methods.
If you stored the overall delivey folder arrangement as an LHA full of empty folders... and then did a series of 7-zips mini archives each with a folder content... then each folder gets its own mini dictionary, just for dealing with system files of that particular file type.
It should cut down on your system resource requirements. You may find two different compressors deal with each delivered system folder competetively.
Even if you stick with just 7 zip, that might be a way forward? What you are attempting is 6:1 compression. Better than that really, because you have to include decempression tools on your one delivering disk, as well as the 6 floppies worth of data.
-
7zip also have command line so can set dictionary to 4K if I want however even at 64K I lose all the gains and then some compared to just using 7z in a single sweep w/o first storing it.
The advantage of XZ is it's the same compression but not using as much memory to decompress, and XZ compressing a "stored" archive is more efficient then 7-Zipping the files (as there's more data to work with).
You should be able to set the dictionary to 256K with XZ and get better compression whilst still being able to extract.
(btw, when I said compress using 7-Zip I actually meant compress using the 7-Zip GUI tool, but using XZ compression... although I realise it doesn't read like that!)
-
It should cut down on your system resource requirements. You may find two different compressors deal with each delivered system folder competetively.
Another optimisation would be to use a BCJ filter. However, there isn't one for 68k so it would need writing... but that should help the compression a fair bit with the executables.
-
Hmmm... might be an idea to try combining compression methods.
If you stored the overall delivey folder arrangement as an LHA full of empty folders... and then did a series of 7-zips mini archives each with a folder content... then each folder gets its own mini dictionary, just for dealing with system files of that particular file type.
It should cut down on your system resource requirements. You may find two different compressors deal with each delivered system folder competetively.
Even if you stick with just 7 zip, that might be a way forward? What you are attempting is 6:1 compression. Better than that really, because you have to include decempression tools on your one delivering disk, as well as the 6 floppies worth of data.
Ya 6:1 compression ratio is never going to happen... I get about 2.5:1 atm. With a bit of clever optimization that gives about 2.4MB on the HDD after installation. What you won't get is A2322 support, HDBackup/BRU, Printers (except Generic), _bullet Fonts and Dan/Esp/Ita/Ned/Nor/Por Locals.
I've split the data into 4 archives, a small boot archive (everything needed to boot and use the contents on the disk), gfx (icons and fonts), txt (Dosdrivers, locales, scripts etc.) and bin (rest). By splitting so similar type of data is together I got a bit better compression anything else I've tried when it comes to store then compress either don't give better compression ratio or cost too much memory to extract.
-
The advantage of XZ is it's the same compression but not using as much memory to decompress, and XZ compressing a "stored" archive is more efficient then 7-Zipping the files (as there's more data to work with).
You should be able to set the dictionary to 256K with XZ and get better compression whilst still being able to extract.
(btw, when I said compress using 7-Zip I actually meant compress using the 7-Zip GUI tool, but using XZ compression... although I realise it doesn't read like that!)
I understood that after a while... XZ refusing to accept the file was a hint... bit I was working with a 6 y/o 7zip that didn't have xz implemented. Good now that's updated. :D
Another optimisation would be to use a BCJ filter. However, there isn't one for 68k so it would need writing... but that should help the compression a fair bit with the executables.
I'm all for better compression with little to no extra memory usage. Tell me when you find a BCJ filter. :D
-
The advantage of XZ is it's the same compression but not using as much memory to decompress, and XZ compressing a "stored" archive is more efficient then 7-Zipping the files (as there's more data to work with).
You should be able to set the dictionary to 256K with XZ and get better compression whilst still being able to extract.
(btw, when I said compress using 7-Zip I actually meant compress using the 7-Zip GUI tool, but using XZ compression... although I realise it doesn't read like that!)
Ok I stand corrected, I was able to gain 17KB (after deducting the the 13KB for shrinkler crunched xzminidec) using store then compress without running out of memory. It does mean I have to up the requirement of HDD space from 3MB to 5MB but that's cool (even back in the days Commodore thought an OS partition should be at least 8MB).
And I'll probably look at a different tool to store the files than 7z now that the compression is taken care of by xz since there's smaller unpackers than 7zdec.
-
I'm all for better compression with little to no extra memory usage. Tell me when you find a BCJ filter. :D
Should be easy enough to write for somebody with knowledge of 68k asm.
PPC version: http://git.tukaani.org/?p=xz.git;a=blob;f=src/liblzma/simple/powerpc.c;h=54dfbf102878ef53ecf694ae3eb7e68473811be1;hb=HEAD
All it does is replace relative offsets in Branch/Call/Jump instructions with absolute values. The theory being that subroutines are referenced multiple times, and this makes the code more repetitive and compressible. The filter than undoes this change when it is decompressed.
And I'll probably look at a different tool to store the files than 7z now that the compression is taken care of by xz since there's smaller unpackers than 7zdec.
If you're doing a store then an Amiga-specific archiver is best as it will keep all the special attributes. LhA would be my suggestion, maybe even go back and find a really old version as it'll be smaller. The newer ones have all sorts of extra compression schemes and things that won't be relevant. :)
-
Should be easy enough to write for somebody with knowledge of 68k asm.
PPC version: http://git.tukaani.org/?p=xz.git;a=blob;f=src/liblzma/simple/powerpc.c;h=54dfbf102878ef53ecf694ae3eb7e68473811be1;hb=HEAD
All it does is replace relative offsets in Branch/Call/Jump instructions with absolute values. The theory being that subroutines are referenced multiple times, and this makes the code more repetitive and compressible. The filter than undoes this change when it is decompressed.
If you're doing a store then an Amiga-specific archiver is best as it will keep all the special attributes. LhA would be my suggestion, maybe even go back and find a really old version as it'll be smaller. The newer ones have all sorts of extra compression schemes and things that won't be relevant. :)
We can always hope someone makes that filter, I can't do it myselv and I'm not holding my breath.
Think I'll go with LZX as I had before... with UnLZX being only 10KB Shrinkler compressed it's hard to beat. :D
-
We can always hope someone makes that filter, I can't do it myselv and I'm not holding my breath.
I might be able to work it out given enough time but I also might screw it up spectacularly. Hence why ideally it needs somebody who knows about 68k asm.
In the meantime I've shaved 5K off xzminidec for you. :D (by recompiling optimising for size and then stripping the binary, hopefully this hasn't broken anything!)
-
I might be able to work it out given enough time but I also might screw it up spectacularly. Hence why ideally it needs somebody who knows about 68k asm.
In the meantime I've shaved 5K off xzminidec for you. :D (by recompiling optimising for size and then stripping the binary, hopefully this hasn't broken anything!)
I'm sure you'd screw it up far less than I would. ;)
Thanks for the slimmer zxminidec. It translated into 624b saving when shrinkler had run through it but every block counts so this helps... if it's not broken.