I have read that there are speed issues with accessing the memory on your motherboard with the A4000. If you have a processor card with RAM on it would be faster, even if it was another 25mHz '040 CPU on that board.
I have an A4000D in a Power Tower case, with fully populated RAM on the motherboard and the stock CBM '040 processor board. Unfortunately, I don't have the cash for the Accelerators available at the Amiga stores online. They all seem to just have the PhaseV boards, ect. And yes, I do run into speed issues myself from time to time. My biggest problem is with running out of RAM though. (Another reason to try to find a processor board, even an older one as long as you can add RAM to it.)
One thing that gave me a bit of speed was adding a Gfx card since this took a lot of load off of the custom chips. I have a PicassoIV which has the built in Flicker-fixer/Scan-Doubler so I don't have to worry about having an older and smaller Amiga moniter hooked up to play the old games.
My System:
A4000D in Power Tower Case
AmigaOS3.9
PicassoIV
X-Surf II (For DSL connection)
A2091 with GuruROM (for my SCSI burner, Z-II RAM disabled on it)
Sony CDU 924S SCSI burner
56x IDE CD ROM
It's a nice system, but it has a lot of room for expansion and improvement.
4gb HD and a 2gb HD