Maybe the 68000 socket is a bit dodgy? Some of these plug-in expansions with thick pins can really knacker the socket.
Also, another thing to check on A500's is that slender piece of shield than runs along the side expansion slot - make sure there's no shorts or anything when the edge connector is in use.