Like others have said, if the A3640 is worth it or not depends on what you're going to be doing. I haven't been doing anything special with my A3000, but I definitely notice when I removed the A3640 for a bit. Things just felt slightly faster with the A3640. You can also try playing with tools such as Blizkick or MMULib to optimize your system even more.
Rev 3.1 of the A3640 solved most of the DMA problems, and rev 3.2 cured the remaining few quirks. It is recommended to use it with KS 3.1. You can't use the rev 3.0 A3640 unless it's been modified into a 3.1 or 3.2. Also, make sure it has a good-sized flat heatsink on the 68040. It tends to run hot in the confined space of the A3000D. You might even want to consider attaching a little CPU cooling fan to it.
-David