Amiga.org
Amiga computer related discussion => General chat about Amiga topics => Topic started by: asrael22 on January 17, 2018, 03:02:34 PM
-
Hi.
I'm trying to archive a large folder (~200MB).
Using LHA it will always stop at some file, telling me that it cannot open it for reading. Depending on what's in the archive it is always a different file.
This is my lha cmdline: "lha -2 -a -e -r -x a ".
What could this be?
(Works fine with LZX.)
Manfred
-
Are you trying this from CLI, SHELL or have tried both?
What have you got your stack set to?
Is this running under a 68060?
Also, shouldn't the command be this way around?:
"lha -2 -a -e -r -x a "
-
Hi.
Yeah, cmd line is the other way around.
The stack was a good pointer. Because there are a _lot_ of files and it seems LHA scans them all before compressing.
So I've increased the stack of the Shell from 4k to 128k. Unfortunately it didn't help.
What's the difference between CLI and Shell?
This is a 68080 (Vampire) CPU.
Manfred
-
To be honest, I forget the difference between CLI and shell, it's been a while since I knew!
EDIT:
AmigaOS.net says this:
"For historical reasons, you will still find references to the CLI in AmigaOS but this is really a shell that is used everywhere in the system. "
I did try and reproduce the problem using WinUAE with A1200 config +8MB fast, with the 68020 version of LhA, and a 231 MB directory of files last night
- it wasn't able to complete either.
I ran out of RAM disk about the time the LhA file got to 84MB
- which was about 3 or 4 hours after I started it!
If you haven't tried already I would try archiving it again but using the 68k version of LHA 2.1.5, and see if that makes a difference - as I appreciate the Vampire is not a true 060, but I have heard that some 060's were better at running 68000 code than they were at running 020, 030, or 040 versions of code.
I'm re-running the test but this time with WinUAE configured for 256MB z3 FAST....
This may take a while!
-
Turns out my laptop paused WinUAE when I opened VirtualBox, and wouldn't come out of it, so I had to restart WinUAE and the LhA!
I will get back to you...
-
Hmm, that doesn't make a lot of sense.
Why would it eat up such a lot of fast ram?
If that really is the case then it's a bug.
I don't think it has something to do with the Vampire. Although I cannot be sure.
I'll try the same thing on a 68060.
Manfred
-
Amiga file systems used a bitmap (for where files are located) that is updated regularly, which I guess could cause a performance hit so I guess the developer of LhA tried to mitigate this using RAM:
- I only had 8MB of RAM: configured when i ran out btw.
LhA still going.. 51 MB so far...
-
create a file in envarc: with the name LHAOPTS
Inside it put only "-b32" but without quotes.
Test again.
-
Hmm, that seems to have worked.
What is that setting?
Manfred
-
I'd love to know the answer to that too!
Meanwhile, my LhA is using 16,777,216 bytes of RAM:
-
Well Manfred,
my LhA job just finished, 231 MB archived to 194MB in a super quick 4 hours and 59 minutes.
How long did yours take again?....
-
That option creates a 32KB IO buffer and will not display any file progress.
It is in its documentation.
-
Hmm, that's interesting. Actually I've looked for some options and have seen -b option.
But I didn't try it because I thought that I/O buffer only sets the general read/write buffer for the files. The documentation does not mention that this setting helps with lots of input files to complete at all. So I'm a little surprised.
Anyway, this setting has helped.
Thanks a lot.
Manfred
-
Well Manfred,
my LhA job just finished, 231 MB archived to 194MB in a super quick 4 hours and 59 minutes.
How long did yours take again?....
256MB compressed to 99MB.
Didn't take the time. Will do again. Currently in operation...
-
256MB compressed to 99MB.
Didn't take the time. Will do again. Currently in operation...
I wasn't at the computer when it finished, but it was well under an hour.
Manfred
-
Hmm, that's interesting. Actually I've looked for some options and have seen -b option.
But I didn't try it because I thought that I/O buffer only sets the general read/write buffer for the files. The documentation does not mention that this setting helps with lots of input files to complete at all. So I'm a little surprised.
Anyway, this setting has helped.
Thanks a lot.
Manfred
Even more curiously, setting a 64k buffer (-b64) doesn't work either.
-
I wasn't at the computer when it finished, but it was well under an hour.
Manfred
Before I went to sleep, I decided to run a comparison againts 7zip on the i5-3320m Windows 7 laptop I was running that WinUAE on.
To compress the same 231MB of data I did using LhA, i used 7z archive format with ultra compression, 64MB dictionary size and 2 out of 4 available CPUs*
It took 2 minutes and 9 seconds.
While it was running, it used 699 MB of RAM - this amount of ram did not fluctuate - and despite the settings in effect it used all 4 cpu cores to perform the archive process.
Now, the way LhA archives appears to be different.
This archives 1 file at a time into temp and THEN appends the data for that one file only to the LhA file it has created. If the data it is reading from is store on disk, this typically has a big overhead on an amiga. If the LhA file it is writing to is tored on disk, this has a massive overhead on an amiga.
So for a comparison i'm going to repeat the process in RAM: alone.
Edit @ 14:59, 15:01:
WinUAE is reading from DH0: (a windows folder) at approx 500 KBps. This is one heck of a bottleneck.