Sounds like you were either not running a well patched AmigaOS. Or all you did was load WB and open a folder marked WHDownload
With patches to move as much stuff as you can to FastRAM, to use the CPU instead of the Blitter, SFS etc. the speed increase is noticeable.
Correct me if I'm wrong, but FastRAM and CPUBlit utilities work with '030, as well. And, since the 3640 offers no on-board memory, there is no speed advantage using a 3640 for these vs. using the standard '030.
Admittedly, I did not test SFS (it wasn't out back then -- this is back in AmigaOS 3.1 days)... FFS performance was not changed, though.
The A3000D in question was primarily a backup and general-use machine. The 3640 did offer better render times for software tuned to use '040 optimizations, but all other general-use operations (desktop/video/word processing) ran about the same.
Nothing very noteworthy, compared to the amazing improvement a Warp Engine or CyberStorm '040 offers.