Faster directory listing possible?

[olsen]

I've got my old 3.9 bb2 installation running again. It's on a 8GB microdrive that has multiple partitions.
One of the partitions contains a LOT of folders with WHDLoad games, each folder has a game in it. When I open this partition in workbench it takes a few minutes before the listing shows. In the meantime I can't do anything else.
This partition is either in FFS of SFS, I don't quite remember, but it takes so long. Is there a way to speed up the directory listing?

Yes. But you would have to change how the data in side that folder is organized by breaking it down into subdrawers, of which each would holds a part of the whole set. You don't have much leverage here: the most effective change you can make is in giving the operating system less work to do.

The source of the performance problems is manyfold.

The file system performance is not too good on AmigaOS, both because of how file operations are being performed by the operating system, but also how poorly the Amiga default file system works.

The way in which Workbench accesses the contents of a directory, figuring out which icons should be displayed, is very complex. Workbench has to collect the names of all icon files in the directory, then load the icons and place them in the drawer window. This may not sound like too much effort, but the way in which this work is being done comes with a large overhead: whenever Workbench finds an icon, it switches to loading that icon, then resumes reading the directory (going back and forth between the two tasks is very costly). While it is scanning, Workbench cannot do anything else, except react to drawer windows being resized or borught to the foreground.

Lastly, if you did not snapshot the icons which Workbench found, additional time will be spent by placing them in the window, so as not to have them overlap with other icons. The process by which the icon position is picked works reasonably well for small numbers of icons, but it quickly becomes slower the more icons are in a drawer.

The deck is stacked against you. All the software layers of the operating system which are involved in eventually showing icons in a Workbench drawer window were originally designed for the small computer system which the Amiga was in 1986. And for this target platform, the software just about worked (arguably, it worked poorly even back then).

What you can do today in order to obtain better performance is in taking measures to make the Amiga do less work. Because the more work it has to do, the more time will be spent: for two times the work, you may see an increase of 4-6 times the time spent on doing it.

I remember from older installations there was a .fastdir file that contained the listing? Never seen that on 3.9 though.

That's because the ".fastdir" files are from an entirely different age, and a different software. They were created by a program called "CLImate", which could be considered one of the many precursors to "Directory Opus" (just to name the most prominent program of its kind). If I remember correctly "CLImate" was released in 1986/1987.

The ".fastdir" files were snapshots of the contents of the respective drawers which they were found in. By reading the ".fastdir" files, "CLImate" could quickly display what was stored in the drawer without going through the lengthy process of scanning it. Scanning a drawer involves reading about 40-50 times the amount of data that is actually going to be used for showing the drawers listing (name, size, date, etc.). Hence the name ".fastdir".

[olsen]

I don't entirely agree with this.

If he opens his window with thousands of icons then Workbench will completely thrash the memory list and there will be thousands of memory fragments and the Amiga will be permanently slower from that point on.  If he opens and closes that window multiple times and then runs random software he is in for a crash.  Its like Workbench doesn't use memory pools at all and just bashes the OS with zillionz of malloc()/free().

Problem is, Workbench and icon.library were designed to work together, but have completely separate programming interfaces. In fact, icon.library has always had a public programming interface, whereas there was no such thing for Workbench until Kickstart 3.x came along.

Because of how Workbench loads icons, it has to talk to icon.library, and the programming interfaces of icon.library are just not up to snuff in terms of efficiency. For example, icon.library performs its own memory management and tracking.

This has to account both for image data and the rest. This is something which you cannot easily switch over to memory pools. For example, you would have to have a separate pool for each Workbench window in order to make pool management more efficient. But Workbench just isn't designed to work that way, and really really hard to change.

As I wrote before: if you want better performance out of the whole stack of software layers which Workbench sits on top of, you should put effort into making the whole stack do less work.

If you try to improve only part of how this whole mess comes together, chances are that any improvements will be consumed by the rest.

[olsen]

I do have a working lightly modified (mostly vasm peephole optimizations) AmigaOS 3.9 workbench.library at 191168 bytes where the original was 200856 bytes. I've been testing it for a while for problems and haven't found any yet. I thought PeterK might want a working complete disassembly which is easier than patching the executable for his modified workbench.library but he was worried about it being more error prone which it is . I don't want to release another version without his patches and I didn't think the code was too bad.

Um, my advice, if you want to hear it, would be not to bother with attacking the workbench.library performance problems in this manner. The problems which cause Workbench to perform so poorly are due to poor choice of algorithms, data structures and overall architecture.

These choices go way back to 1985/1986 when the original Workbench was designed, and subsequent reworkings/enhancements made to this code did not resolve the limitations of the original design. If anything, they compounded the problems.

Workbench and its partner in crime icon.library are very closely-coupled and share dependencies which are spread through the entire Workbench code. These dependencies are what make it hard to change either part of the couple. Because Workbench and icon.library have to work under modest memory conditions, and with legacy software, there is little room for making larger changes. So this is why necessary architectural changes have rarely been implemented, if at all.

Looking at Workbench through the lens of 68k assembly language, and the optimizations that are possible at this level, will likely give you only a tiny picture of what makes Workbench tick, and how its chief problems look like (or even how these might be addressed). There is likely nothing you could do that will have a measurable effect on Workbench performance, no matter how much effort you put into it.

There is a reason why the long list of maintainers of the Workbench code each advocated throwing the code away and starting over from scratch.

[olsen]

No doubt. The original reason for a new workbench.library was to fix a few bugs.

No, not exactly. The original reason for updating workbench.library was to produce something which would make AmigaOS 3.5 a more attractive product. As Workbench is the "face" of AmigaOS, changes to make it more responsive (e.g. real-time scrolling) and more attractive (e.g. colourful icons) were at the top of the list. As for bugs, there were not that many to take care of. The code, architectural burdens aside, was in remarkably good shape.

The other half of the necessary changes happened to the workbench.library API, which was opened up, and also supplemented by an ARexx interface.

A simple cleanup and peephole optimizations that saves 10k is just a bonus and not an attempt to fix the AmigaOS 3.9 bloat :/.

Bloat? Well, the most work that went into Workbench itself was intended to open up the APIs, integrate the new icon.library functionality, make Workbench more responsive and of course replace the built-in "Information..." requester.

I wrote the new "Information..." requester from scratch, and it's this part which made workbench.library much larger than it previously used to be.

For the 3.5/3.9 updates we switched from Lattice 'C' 5.04 to SAS/C 6.55, which should have produced better code. But as the optimizer was set for speed (rather than size, stun or kill) the overall size of workbench.library went up rather than down.

Have you tried PeterK's icon.library? It gives a remarkable speedup (including low end systems) without changing the interface. There is a lot possible with assembler but it is a lot of work. It shouldn't be necessary if compilers were doing a better job.

No, I can't say I did. If I never have to see workbench.library or icon.library from the inside I won't regret it I did my part in updating both (I rewrote icon.library from scratch for the 3.5 update), but like so many programmers before me, I was unable to address the architectural limitations of either. There is only so much you can do if the Workbench design is actively playing against you...

[olsen]

The way I figured it, if Workbench used its own pool and PeterK's icon.library used its own pool then that could double the speed and/or reduce memory fragmentation by 10.  Or something like that

No, Workbench does not use memory pools. It pulls everything from the globally available memory, which leads to fragmentation.

I assume this is why PeterK's icon.library is so much faster (+ all the other optimizations he did.)

I don't know how it works, but there is only so much that can be done, given how Workbench reads icons. What breaks the camel's back is that icons are stored in individual files which must be found first and read quickly. Workbench tries to do that by scanning directory contents and switching back and forth between scanning and reading individual icons, which causes disk accesses to jump around wildly.

If this process could be sped up, it would have to be smarter about when to read the icons. For example, if you could cache the icons to be read from a directory then this would reduce the amount of time which Workbench spends on switching between scanning the directory and reading the icons.

Let's say you had an icon cache. If you could tell Workbench to hold off reading the icons until it has finished scanning the directory, then you could cut down the overall time spent by a significant amount. Of course, once Workbench has read the directory contents, reading the icons must be very quick. I think all of this would be doable.

[olsen]

I always "just assumed" that the reason PFS3 was so fast was because it essentially had a secret built-in copy of FastCache.

Is that not true in some sense?

Caching is helpful, but it is not essential. A cache will boost performance only if what you need to access next is already stored in the cache. This is the case with data which is accessed repeatedly, but often enough data on the disk is accessed once and then never again.

To provide for a consistent speed improvement you need to organize the file system's data structures in such a way as make accessing the data fast. For example, instead of having to walk through a series of data blocks before you can use the data you are looking for, you could have a tree data structure, which would let you pinpoint within 3-4 steps what would take 15-30 data blocks to walk through in sequence.

The FFS consistently uses data structures which are very simplistic in construction, and in theory ought to be robust. But the robustness is just not there, and the simplicity only succeeds in dropping performance the more data has to be managed.

Ppl always complain about FFS being slow but if you increase the sector size to 4K or larger you get a giant speedboost.  No caching software needed.  If you then add caching software you should then be around the same speed as SFS or PFS.

Changing the block size (a block is made up of several sectors) will have a positive impact on performance only if your disk drive and the controller hardware do not make data accesses noticeably slower if you end up reading 4-8 times the previous amount of data.

You still have the same scalability issues with the FFS, which means that the more data you have to manage, the quicker FFS loses performance. Not all file systems lose performance as quickly as the FFS, as the amount of data increases. It's particularly nasty with the FFS, though.

There is one major drawback to increasing the block size with the FFS: if the file system data structures become corrupted, then the amount of data affected by the corruption will be much higher than with smaller block sizes. Given that the FFS lacks robustness by design, trading greater speed for even smaller robustness may be a bad idea...

[olsen]

I have addbuffers Work: 1000 on my 32MB FFS A1200

1000 buffers = 512k

Addbuffers does speed things up noticeably.

But it is not as dramatic as FastCache or PowerCache or etc.

The type of buffer which the file system manages through "AddBuffers" is only used for file system data structures. The contents of the files are not buffered in this manner.

It's possible that you can obtain the same performance with a lot less buffers thrown at the file system to manage. The old FFS has to walk through the entire list of buffers before it can find the one it is looking for.

As such, the more buffers you throw at the file system, the more time it will spend on managing the buffers, rather than making good use of what is in the buffer.

[olsen]

Should I use like only FastCache or FastCache and AddBuffers together?

Because the file system is very limited in what it can do with the buffers assigned to it, you might be best served with a block caching solution (FastCache) doing the hard work, and the file system using only the bare minimum of buffers it needs (10-15 buffers).