Hello-
There's two problems you could be running into here, so I have a couple questions.
First off, how many files is "a sizable number"? 100? 1,000? 10,000?
How are you opening the directory? (Workbench, DirOPUS/DM, etc?) If it's workbench you're using... It has to allocate the graphics for all the icons out of chip memory. (And even though all the icons may be the same, each takes up a chunk of memory... chip memory, if you don't have a graphics card.)
If you're using DirOPUS or something, that shouldn't be a problem, though. But, it still has to keep all those file names in a buffer of some sort. Try a directory caching program. It should help in that case. There is a good one on Aminet, but I forget it's name, as I haven't messed with my settings in a long time...