You can't switch the "daughterboards". The A3000 has the CPU and FPU soldered onto the motherboard directly. AFAIK, only some of the A3000T's and the prototype (A3000's that had the DSP and AGA on them) didn't have the CPU on the motherboard. Although you might have to check it out for yourself.
The one big problem with the A3000 is that you need to take the whole thing apart to get to the ram, cpu slot, buster, ROM chips.
The A3000 also uses ZIP chips for ram, the A4000 has SIMM slots. ZIPs are a pain to find, a pain to put in. Both systems top up at 16 megs fast so you'll need to look at an accelerator board with SIMM slots on them if you want more. (The 3640 houses only the CPU, nothing else.)