It's probably the flash drive. If it takes too long to fetch information from the .wad file it will wait for it to load until continuing. Also, if it's loading code to the Chip RAM that will slow it down too. If you have a real IDE hard drive sitting around or a RAM expander that will go up to 32 MiB or more, they will both help.
For the 32 MiB to help, just rename the .WAD file to something else, make a softlink from the Flash Drive to the RAM disk and copy the .WAD file to the RAM disk every time you start the game. If you've done it right, you shouldn't need to access the Flash drive except for save games.
