I guess it´s because you open the directory in a window, in workbench. All the contents of the directory will consume chip-ram to, because of the window bitmap...
You could avoid this by using a CLI instead. KingCON: as a great alternative which has directory listing using the TAB key... Most file managers as DirectoryOpus are also alternatives to window browsing...