a question about block and buffer sizes in HDToolBox with OS3.1.4

[klx300r]

so I've got the latest OS3.1.4 installed and happy in my A1200 but when I installed it I quickly flew through the HD install process and left the block size at 512kb and buffers at 30. I don't have a clue with the directSCSI tick box is so I left it blank.
QUESTIONS: I have a 16GB DOM working fine with FFS now BUT

1- should I change the block size to 4096? (I know I'll have to load up a ton of stuff already on it if I do)
2- should I change the buffer size to more than 30?

[guest11527]

so I've got the latest OS3.1.4 installed and happy in my A1200 but when I installed it I quickly flew through the HD install process and left the block size at 512kb and buffers at 30. I don't have a clue with the directSCSI tick box is so I left it blank.

That's probably 512 bytes (not KByte). Are you using the on-board IDE of the A1200? If so, DirectSCSI is not required, just leave it blank.

1- should I change the block size to 4096? (I know I'll have to load up a ton of stuff already on it if I do)

If everything works to your liking, just keep things as they are. Otherwise, 4096byte blocks allow the FFS to verify the disk faster, i.e. bootstrap will be faster.
If you change the block size, you need to reformat the partition.

2- should I change the buffer size to more than 30?

If you have the RAM, do. You can still do that in the startup-sequence, i.e.

Code: [Select]

addbuffers dh0: 200
provided your partition name is dh0:

[klx300r]

Thanks for the quick answers Thomas much appreciated!
for question 2 do you mean chip or fast? if fast i've got plenty (32MB).  Also what determines the optimum buffer size?

[Gulliver]

Also what determines the optimum buffer size?

I wrote this for the 3.2 FAQ, it also applies for you too.

4.8 * How can I speed up disk operations?

We already have a cache system built-in that it is severely
underused. That is the case of buffers that can be set within
HDToolBox in a non destructive manner, or by using the AddBuffers
command.

Nobody usually tweaks this values, and this is a huge performance
boost possibility that often gets overlooked.

Buffers are file system independent, so they should be safe to be used
under any AmigaOS compatible file system.

This is a rough guide that will attempt to provide you with advice on
how much memory should be destined for buffer usage, so take it with a
grain of salt. It really depends on your Amiga habits and your
hardware configuration. So don't be afraid to tweak these values and
experiment in order to achieve the best results.

For any drive, remember that each partition in your system will have
buffers by default assigned to it, so first check what amount they use.
And also consider that all those partitions, no matter which one is
bootable, will add their buffer usage to generate a grand total.

Yes, buffers are great, but they are certainly not for free, as they
consume ram.

Buffers only make sense when the data to be cached will be accessed
several times, and for that to happen they need to be preloaded with
something to start caching. So the idea is that the system will get
much faster from the second time onwards that it accesses the same
data.

The following is the total amount of buffer ram usage we recommended
(it should be divided between all your partitions). Note that all
values are approximate.

Total system ram  -  Recommended amount of ram to add to buffers
  2 MB            -  Don't add more buffers, you can barely run the OS!
  4 MB            -  128 KB (256 buffers)
  8 MB            -  256 KB (512 buffers)
 16 MB            -  512 KB (1024 buffers)
 32 MB            -  1 MB (2048 buffers)
 64 MB            -  2 MB (4096 buffers)
128 MB            -  4 MB (8192 buffers)
256 MB            -  8 MB (16384 buffers)

To apply a change to the system buffers just add the corresponding
buffer value to the volume/s you wish. The easiest method is by simply
modifying the startup-sequence in the AddBuffers line.   

[nbache]

Am I right in remembering that these buffers are caching file system data blocks as opposed to actual file data blocks? So the effect of adding more is supposedly greater for volumes with many relatively small files, and/or many files in each directory, rather than with fewer, bigger files?

If so, maybe this aspect would be useful to mention in the otherwise very good FAQ entry.

Best regards,

Niels

[Gulliver]

Am I right in remembering that these buffers are caching file system data blocks as opposed to actual file data blocks? So the effect of adding more is supposedly greater for volumes with many relatively small files, and/or many files in each directory, rather than with fewer, bigger files?

If so, maybe this aspect would be useful to mention in the otherwise very good FAQ entry.

Best regards,

Niels

You are absolutely correct on what you just posted.

I will integrate your comments to the answer in the FAQ.

Thank you!

Best regards,
Ignacio

[guest11527]

Am I right in remembering that these buffers are caching file system data blocks as opposed to actual file data blocks?

Almost. The buffers are used for two purposes: First, as you say, to store metadata on the files, that is, administration information. Second, if the data buffer the user passed to the file system does not fit to the mask, the file system also has to allocate a buffer. It then reads data first to the buffer, and then copies the data from the buffer to the target destination where the user requested it. This also happens if the user requested data does not fill an entire number of sectors, and some "fractional sector" needs to be transfered.

So the effect of adding more is supposedly greater for volumes with many relatively small files, and/or many files in each directory, rather than with fewer, bigger files?

It is not quite that simple. The meta information is not only directory information (how is the file named), but also "file list blocks", that contain information where on disk the file is located. If files are large, and you seek within the file a lot, then it helps a lot to have these file list blocks cached. A typical use case is that of "disk images" for emulators. Having insufficient buffers then delays reading such "hard-files" quite a lot.

Unlike what Gulliver said, I wouldn't go with the number of buffers beyond a reasonable number, e.g. 256, unless you really have to (see above, for hardfiles). More buffers are not necessarily faster, because it also means that the FFS needs to search in a larger set to find the right block, and this takes longer if there are more blocks.

[Gulliver]

Unlike what Gulliver said, I wouldn't go with the number of buffers beyond a reasonable number, e.g. 256, unless you really have to (see above, for hardfiles). More buffers are not necessarily faster, because it also means that the FFS needs to search in a larger set to find the right block, and this takes longer if there are more blocks.

Well, it is a matter of usage, that is why I mention to test and
experiment with those values.

I will give you an example of the configuration of a high-end Amiga
user that I visited the other week. His system is an Amiga 3000
equipped with a 68060, 256 MB of fast memory and a Mediator with
network, gfx, sound and usb pci cards.

He mostly uses it to do some 3d rendering stuff, play mpg videos
and 3d gaming (we played Quake).

We did experiment with buffer settings and he got a great performance
increase on his usual activities by allocating a total of about 4000
buffers (counting all partitions and devices present). Beyond that,
improvements were not really noticeable and seemed a waste of memory.
So he settled with that.

A question: What is the algorythm used to discard cached data when buffers are full? LRU? are they just FIFO buffers? or do they follow other algorithm?

[olsen]

Unlike what Gulliver said, I wouldn't go with the number of buffers beyond a reasonable number, e.g. 256, unless you really have to (see above, for hardfiles). More buffers are not necessarily faster, because it also means that the FFS needs to search in a larger set to find the right block, and this takes longer if there are more blocks.

And that is precisely the problem: in the FFS (in Workbench/Kickstart 1.3 of 1987 and beyond) and also its precursors (Workbench/Kickstart 1.1 and 1.2 of 1985/1986) the "buffers" managed by the AddBuffers command are not really a cache

A cache only has value if it already contains data that is likely to be retrieved next. For the Amiga ROM file system that seems to hold true only for a handful of blocks, namely the root block, the bitmap blocks, the metadata blocks of a file or directory currently being scanned, and that's mostly it (for the OFS variant, which has data blocks with checksums, these go into buffers, too).

Data in those buffers is not intended to stick around and age, with more frequently-used data lasting longer than only briefly-used data. The file system may notice if a block that's needed is already present in the buffers, but it's more a convenience than a goal to create such lucky accidents. The buffer management is intended to provide temporary storage on demand when reading or writing, and whatever is available is pulled off the rack and put to use. Never mind if the next disk access could have been avoided if what has just been reused would have stuck around a bit. These file system buffers are "bounce buffers", not cache buffers.

Another unfortunate clue that this is not supposed to be a cache is that the file system becomes slower in retrieving a "useful" block from the buffer because the currently unused buffers are stored in a linked list. Every access has to check the entire list (at about O(n^2) complexity, everything considered).

So, with the 68k FFS and its precursors, it is good advice to keep the number of buffers small because larger numbers will slow things down noticeably. The design aspect which produced these side-effects probably mattered little at the time it was created, when the mass storage device of the day was the floppy disk (perfectly fine with 10 buffers) or the 20 MByte hard disk drive (perfectly fine with 20 buffers).

Side-note: because the FFS reimplementation for AmigaOS4 (and MorphOS) was a rewrite from scratch, its built-in buffers do act as a cache (with about O(log(n)) complexity), and it has dedicated seperate bounce buffers for the rare cases when these are needed.

[olsen]

A question: What is the algorythm used to discard cached data when buffers are full? LRU? are they just FIFO buffers? or do they follow other algorithm?

The algorithm, as far as I can tell, knows no purge conditions and has no deliberate replacement scheme. Its goal is to find a buffer which is not currently being used for a read or write operation in progress and put it to use. If data is to be read, it will make an attempt to check if the data to be read is still sitting in an used buffer (the more buffers it has to check, the slower it becomes; you got lucky that 4000 buffers worked for you). If it's not sitting there, then it will make do with the next conveniently available buffer.

This is not a cache.

[Gulliver]

Thank you very much both Thomas and Olaf.

I overestimated the use of buffers and now clearly understand that their
use as a sort of cache is incidental not an intended goal. And they are
not really efficient at that either.

A more conservative approach on their usage sounds as better advice.

[10MARC]

While I appreciate Gulliver's guide as to the recommended number of buffers, it does not seem to mention that each buffer size increases as your block size increases. I am sure that I have seen this. If you use a 4k block size, each buffer is now 4k and if you use a large number of buffers, RAM usage goes through the roof. I think Gulliver's list is accurate for 512 byte block size, but should be divided accordingly as your block size increases.

[guest11527]

This is not a cache.

Well, it is, just not a very well organized one. If it would be a bounce-buffer, it would be only used if the mask does not fit, but that's not quite how it works. It also caches metadata blocks that have been read recently, and that's why it is helpful for hardfiles, for example. Probably not as helpful as it could be, but yes, to some degree.

At some point, I should really replace the simple hash list it currently is to allow larger sizes and make it more efficient at larger sizes. But then again, I don't dare to touch it if isn't broken (enough), at least not at the moment as there is really a lot more pressing business to do.

[klx300r]

thanks to all that responded...great info!
I think i'll just leave my buffers in HDToolBox at the default 30 then 

[kolla]

Just a slightly off-topic heads-up - if you use PFS (like pfs3aio for example), the default 30 translates to 150.
http://eab.abime.net/showpost.php?p=1227879&postcount=205