Weird. I suppose your suggestion would account for why it seems to work when its installed in the switcher with the "512k" jumper enabled though... I just wish it would actually boot - it totally ignores any bootable disks I insert (I can hear the floppy drive periodically checking though), even when I've held both mouse buttons down and manually selected the floppy drive as the first boot device.
Try swapping the two CIA chips round and see if anything changes in 1.3 & 3.1.
Kickstart 1.x and 2.x+ use the CIA differently, and while the chips are supposed to be identical, it is possible that you have a damaged one that works enough for 1.3 but not 3.1. In which case swapping them may have some effect. It's possible that it's another chip but CIA's are a good starting point because you are guaranteed to have two of them.
I'd also try using the 3.1 without the switcher by adding the wire, in case it's causing the problem.