Do you have a 68040 or better ? Then it could still be a software problem. The original Workbench 3.1 disk does not contain a 68040.library and thus runs the 68040 (or 68060) in a slow compatible mode. Once you install a 68040.library, this becomes active upon boot and accelerates the processor (enables caches etc.)
Now, if you run a 68060 without the correct 68040.library (dummy) and 68060.library installed, it is a wonder that you get a desktop at all. Or if you run a non-Commodore 68040 card with the Commodore 68040.library you might get trouble, too.
Bye,
Thomas