@jose
How many files do you have in that directory? More than 1000? In such case, the slow-down is inevitable. You need to distribute the files to more than 1 directory.
How many buffers do you have assigned to the partition where this directory resides? I suspect that they are not enough. Increase them to 512 or even more.
You can also increase the SFS read-ahead buffers by using the SFSCache command and see if things get better.
Also, you need to make sure that the the Mask value of this partition points to a location in the Fast-ram.