I think there's something else going on. I have a 2632 as well and it is VERY unstable as soon as I add the memory to the system with the 2632 command, often resulting in the symptom you describe.
We had another thread on here a while back where someone reported an old fix from Dave Haynie for a possibly-related DMA problem that involved adding a resistor between pins on the CPU slot. Or was it the 68000 socket? Either way, the details were too vague for me to attempt it. Anyone know more about this?