Welcome, Guest. Please login or register.

Author Topic: Faster directory listing possible?  (Read 13700 times)

Description:

0 Members and 1 Guest are viewing this topic.

Offline Lurch

  • Lifetime Member
  • Hero Member
  • *****
  • Join Date: Dec 2003
  • Posts: 1716
    • Show only replies by Lurch
Re: Faster directory listing possible?
« Reply #14 on: June 15, 2013, 10:40:14 PM »
Windows XP is bad for doing this. Doesn't happen that often on Windows 7 unless a program has crashed. Programs that do that tend to be 3rd party badly written programs.

Most cases that I see is either the above, a badly written driver or 7 is just not setup correctly.

Have two server boxes here one with Windows 7 which has been up and running a few months now without needing a restart.

The other is Server 2008 R2 which is solid a rock (can even disable the gui if you wish)

Windows 8 doesn't seem to suffer from it, I've been running it for over year now. The only thing I dislike is the new task manager (But wont get into that).

Linux, now here's an OS that I can't kill. Had one server running for years without needing to reboot it and that's using x windows not just the shell.
-=[LurcH]=-
A500 Plus Black 030@40MHz 128MB | A1200T 060@80MHz 320MB | Pegasos II G4@1GHz 1GB  | Amiga Future Sub
 

Offline freqmax

  • Hero Member
  • *****
  • Join Date: Mar 2006
  • Posts: 2179
    • Show only replies by freqmax
Re: Faster directory listing possible?
« Reply #15 on: June 15, 2013, 11:11:55 PM »
@ChaosLord, My curiousity was as to how it prevent this fragmentation?

I think I found the answer here:
http://www.platon42.de/files/util/TLSFMem.readme
« Last Edit: June 15, 2013, 11:19:08 PM by freqmax »
 

Offline nicholas

Re: Faster directory listing possible?
« Reply #16 on: June 15, 2013, 11:27:54 PM »
Installing matthey's CopyMem from Aminet will speed up your 060 machine a great deal too.
“Een rezhim-i eshghalgar-i Quds bayad az sahneh-i ruzgar mahv shaved.” - Imam Ayatollah Sayyed  Ruhollah Khomeini
 

Offline matthey

  • Hero Member
  • *****
  • Join Date: Aug 2007
  • Posts: 1294
    • Show only replies by matthey
Re: Faster directory listing possible?
« Reply #17 on: June 16, 2013, 12:14:18 AM »
Quote from: ChaosLord;737965

If he opens his window with thousands of icons then Workbench will completely thrash the memory list and there will be thousands of memory fragments and the Amiga will be permanently slower from that point on.  If he opens and closes that window multiple times and then runs random software he is in for a crash.  Its like Workbench doesn't use memory pools at all and just bashes the OS with zillionz of malloc()/free().


I snooped the memory allocation functions and it looks like some of the allocations use memory pools and some don't so TLSFmem should provide some speedup. AllocMem() and AllocVec() are probably faster than TLSFmem until the memory becomes fragmented. Some programs really do thrash memory like you say. TLSFmem uses the BFFFO instruction which isn't exactly fast (and slow relative to other instructions on the 68060). Memory pools are another solution for reducing memory fragmentation but I expect they are slower too. A whole pool can be released at once which is an advantage though. It's nice for memory cleanup. Vbcc compiled programs use memory pools for malloc()/free() making the cleanup easy and guarding against memory that did got get deallocated.

Quote from: nicholas;737973
Installing matthey's CopyMem from Aminet will speed up your 060 machine a great deal too.


It doesn't appear that Workbench, the workbench.library or the icon.library are using the exec/CopyMem() functions while listing icons in a workbench window. My commodity.library used CopyMem() with a 16 byte size several times while listing though. Any speedup from my CopyMem patch would probably be minor on a fast processor like the 060 or 040 in this case. Every little bit helps though :).
 

Offline nicholas

Re: Faster directory listing possible?
« Reply #18 on: June 16, 2013, 12:26:21 AM »
Ooh, I wasn't aware you'd done a commodity.library too.  I'll have to grab myself a copy of that! :)

What's your opinion on the PatchFor020 tool on Aminet?  Is it worth running over all the executables on an 3.9 install? I have a CSMK2 060@50MHz.
“Een rezhim-i eshghalgar-i Quds bayad az sahneh-i ruzgar mahv shaved.” - Imam Ayatollah Sayyed  Ruhollah Khomeini
 

Offline matthey

  • Hero Member
  • *****
  • Join Date: Aug 2007
  • Posts: 1294
    • Show only replies by matthey
Re: Faster directory listing possible?
« Reply #19 on: June 16, 2013, 03:00:35 AM »
Quote from: nicholas;737978
Ooh, I wasn't aware you'd done a commodity.library too.  I'll have to grab myself a copy of that! :)

I haven't. I meant "my" unmodified commodity.library on "my" Amiga. Sorry for the confusion.

Quote from: nicholas;737978
What's your opinion on the PatchFor020 tool on Aminet?  Is it worth running over all the executables on an 3.9 install? I have a CSMK2 060@50MHz.

PatchFor020 works well most of the time but is limited in what it patches. It only patches compiler math functions to faster ones. It will patch with 64 bit integer instructions that are trapped on the 68060 making the code slower for the 68060! I've seen a bug in one of the patches where a signed multiply was used when an unsigned should have been. Most of the time this would not cause a problem but could completely fail in rare cases. Make a backup before you apply the patch and test the speed, output and for MuForce/Enforcer hits. There are a lot more 68020 optimizations that can be done than math routines although these offer some of the biggest gains and are big enough that finding and patching them is reasonably safe. Hand optimizing is slow and error prone but has the potential for a much greater speedup and memory savings. Sometimes 2x the speed and 1/2 the size is possible where the compiler wasn't very good.
 

Offline freqmax

  • Hero Member
  • *****
  • Join Date: Mar 2006
  • Posts: 2179
    • Show only replies by freqmax
Re: Faster directory listing possible?
« Reply #20 on: June 16, 2013, 03:25:22 AM »
Alternative approach.. patch workbench icon-hell? ;)
 

Offline matthey

  • Hero Member
  • *****
  • Join Date: Aug 2007
  • Posts: 1294
    • Show only replies by matthey
Re: Faster directory listing possible?
« Reply #21 on: June 16, 2013, 04:22:45 AM »
Quote from: freqmax;737987
Alternative approach.. patch workbench icon-hell? ;)


I do have a working lightly modified (mostly vasm peephole optimizations) AmigaOS 3.9 workbench.library at 191168 bytes where the original was 200856 bytes. I've been testing it for a while for problems and haven't found any yet. I thought PeterK might want a working complete disassembly which is easier than patching the executable for his modified workbench.library but he was worried about it being more error prone which it is :). I don't want to release another version without his patches and I didn't think the code was too bad.

I'm currently working on trying to improve vbbc. It's much better to have the compiler do the work better to begin with. I also am looking at possibly optimizing some AROS libraries and trying to incorporate them with AmigaOS 3.9.
 

Offline JimDrew

  • Lifetime Member
  • Full Member
  • ***
  • Join Date: Jun 2012
  • Posts: 241
    • Show only replies by JimDrew
Re: Faster directory listing possible?
« Reply #22 on: June 16, 2013, 06:23:38 AM »
Have you tried installing any type of device caching software?  Single block reads (which are common when parsing the directory) is extremely slow with flash card type devices.
 

Offline ChaosLord

  • Hero Member
  • *****
  • Join Date: Nov 2003
  • Posts: 2608
    • Show only replies by ChaosLord
    • http://totalchaoseng.dbv.pl/news.php
Re: Faster directory listing possible?
« Reply #23 on: June 16, 2013, 07:03:07 AM »
Quote from: JimDrew;737991
Have you tried installing any type of device caching software?  Single block reads (which are common when parsing the directory) is extremely slow with flash card type devices.


Hey that is a very good point!  When reading from a flash device the device can only read whole blocks at a time right?  And 1 block of that drive is probably like 1MB, right?  Even if it is only 512K that is still going to be really tremendously slow for that crummy IDE v1.0 interface that Mehdi Ali stuck us with.

SCSI 4ever! :)
Wanna try a wonderfull strategy game with lots of handdrawn anims,
Magic Spells and Monsters, Incredible playability and lastability,
English speech, etc. Total Chaos AGA
 

Offline psxphill

Re: Faster directory listing possible?
« Reply #24 on: June 16, 2013, 10:45:03 AM »
Quote from: ChaosLord;737993
When reading from a flash device the device can only read whole blocks at a time right? And 1 block of that drive is probably like 1MB, right?

No, you only have to read 512 bytes at a time (or on new sata drives it's 4k).
 
addbuffers gives more buffers to the filesystem, if that buffering is fast then adding it to the device as well is not worth it.
 
FFS gets slower if you add a lot of buffers & someone did do some device patching software that allowed you put buffering at the device layer.
 
My understanding is that SFS/PFS don't suffer from this.
 

Offline Robert17

  • Hero Member
  • *****
  • Join Date: Apr 2004
  • Posts: 1011
    • Show only replies by Robert17
Re: Faster directory listing possible?
« Reply #25 on: June 16, 2013, 11:44:10 AM »
If it's just WHDLoad listing that are slow due to amoint, you could use a frontend called iGame, it lists all games after scanning a chosen directory. I've used iGame on 030 and 060 - It's fast on both :)

Robert.
Member of the Lincs Amiga Group, UK :-)
 

Offline olsen

Re: Faster directory listing possible?
« Reply #26 on: June 16, 2013, 12:07:14 PM »
Quote from: djkoelkast;737947
I've got my old 3.9 bb2 installation running again. It's on a 8GB microdrive that has multiple partitions.
One of the partitions contains a LOT of folders with WHDLoad games, each folder has a game in it. When I open this partition in workbench it takes a few minutes before the listing shows. In the meantime I can't do anything else.
This partition is either in FFS of SFS, I don't quite remember, but it takes so long. Is there a way to speed up the directory listing?

Yes. But you would have to change how the data in side that folder is organized by breaking it down into subdrawers, of which each would holds a part of the whole set. You don't have much leverage here: the most effective change you can make is in giving the operating system less work to do.

The source of the performance problems is manyfold.

The file system performance is not too good on AmigaOS, both because of how file operations are being performed by the operating system, but also how poorly the Amiga default file system works.

The way in which Workbench accesses the contents of a directory, figuring out which icons should be displayed, is very complex. Workbench has to collect the names of all icon files in the directory, then load the icons and place them in the drawer window. This may not sound like too much effort, but the way in which this work is being done comes with a large overhead: whenever Workbench finds an icon, it switches to loading that icon, then resumes reading the directory (going back and forth between the two tasks is very costly). While it is scanning, Workbench cannot do anything else, except react to drawer windows being resized or borught to the foreground.

Lastly, if you did not snapshot the icons which Workbench found, additional time will be spent by placing them in the window, so as not to have them overlap with other icons. The process by which the icon position is picked works reasonably well for small numbers of icons, but it quickly becomes slower the more icons are in a drawer.

The deck is stacked against you. All the software layers of the operating system which are involved in eventually showing icons in a Workbench drawer window were originally designed for the small computer system which the Amiga was in 1986. And for this target platform, the software just about worked (arguably, it worked poorly even back then).

What you can do today in order to obtain better performance is in taking measures to make the Amiga do less work. Because the more work it has to do, the more time will be spent: for two times the work, you may see an increase of 4-6 times the time spent on doing it.

Quote
I remember from older installations there was a .fastdir file that contained the listing? Never seen that on 3.9 though.

That's because the ".fastdir" files are from an entirely different age, and a different software. They were created by a program called "CLImate", which could be considered one of the many precursors to "Directory Opus" (just to name the most prominent program of its kind). If I remember correctly "CLImate" was released in 1986/1987.

The ".fastdir" files were snapshots of the contents of the respective drawers which they were found in. By reading the ".fastdir" files, "CLImate" could quickly display what was stored in the drawer without going through the lengthy process of scanning it. Scanning a drawer involves reading about 40-50 times the amount of data that is actually going to be used for showing the drawers listing (name, size, date, etc.). Hence the name ".fastdir".
« Last Edit: June 16, 2013, 12:44:42 PM by olsen »
 

Offline olsen

Re: Faster directory listing possible?
« Reply #27 on: June 16, 2013, 12:23:32 PM »
Quote from: ChaosLord;737965
I don't entirely agree with this.

If he opens his window with thousands of icons then Workbench will completely thrash the memory list and there will be thousands of memory fragments and the Amiga will be permanently slower from that point on.  If he opens and closes that window multiple times and then runs random software he is in for a crash.  Its like Workbench doesn't use memory pools at all and just bashes the OS with zillionz of malloc()/free().


Problem is, Workbench and icon.library were designed to work together, but have completely separate programming interfaces. In fact, icon.library has always had a public programming interface, whereas there was no such thing for Workbench until Kickstart 3.x came along.

Because of how Workbench loads icons, it has to talk to icon.library, and the programming interfaces of icon.library are just not up to snuff in terms of efficiency. For example, icon.library performs its own memory management and tracking.

This has to account both for image data and the rest. This is something which you cannot easily switch over to memory pools. For example, you would have to have a separate pool for each Workbench window in order to make pool management more efficient. But Workbench just isn't designed to work that way, and really really hard to change.

As I wrote before: if you want better performance out of the whole stack of software layers which Workbench sits on top of, you should put effort into making the whole stack do less work.

If you try to improve only part of how this whole mess comes together, chances are that any improvements will be consumed by the rest.
 

Offline olsen

Re: Faster directory listing possible?
« Reply #28 on: June 16, 2013, 12:41:42 PM »
Quote from: matthey;737989
I do have a working lightly modified (mostly vasm peephole optimizations) AmigaOS 3.9 workbench.library at 191168 bytes where the original was 200856 bytes. I've been testing it for a while for problems and haven't found any yet. I thought PeterK might want a working complete disassembly which is easier than patching the executable for his modified workbench.library but he was worried about it being more error prone which it is :). I don't want to release another version without his patches and I didn't think the code was too bad.


Um, my advice, if you want to hear it, would be not to bother with attacking the workbench.library performance problems in this manner. The problems which cause Workbench to perform so poorly are due to poor choice of algorithms, data structures and overall architecture.

These choices go way back to 1985/1986 when the original Workbench was designed, and subsequent reworkings/enhancements made to this code did not resolve the limitations of the original design. If anything, they compounded the problems.

Workbench and its partner in crime icon.library are very closely-coupled and share dependencies which are spread through the entire Workbench code. These dependencies are what make it hard to change either part of the couple. Because Workbench and icon.library have to work under modest memory conditions, and with legacy software, there is little room for making larger changes. So this is why necessary architectural changes have rarely been implemented, if at all.

Looking at Workbench through the lens of 68k assembly language, and the optimizations that are possible at this level, will likely give you only a tiny picture of what makes Workbench tick, and how its chief problems look like (or even how these might be addressed). There is likely nothing you could do that will have a measurable effect on Workbench performance, no matter how much effort you put into it.

There is a reason why the long list of maintainers of the Workbench code each advocated throwing the code away and starting over from scratch.
 

Offline djkoelkastTopic starter

  • Full Member
  • ***
  • Join Date: Jul 2004
  • Posts: 200
    • Show only replies by djkoelkast
    • http://www.retroforum.nl
Re: Faster directory listing possible?
« Reply #29 from previous page: June 16, 2013, 12:48:50 PM »
Save images to memory was already enabled, I'll try the icon.library next for as far as I can work with a tft monitor that tells me NOT OPTIMAL RESOLUTION every 2 minutes.
Amiga 4000/060 cybervision64, CyberSCSI MKII, AlfaData BSC MFC3 I/O, Ariadne II, OS 3.9(bb2), 2x IDE > CF 8GB Seagate Microdrive, 1x HD FDD, 1x SCSI ZIP 100

http://www.retroforum.nl