It sounds like you're doing the right thing in terms of disabling the onboard ROM and putting the driver in SYS:Expansion on your boot floppy.
But those card readers use SCSI LUNs, a method of getting multiple SCSI devices at a single SCSI address. i.e., each slot on that card is a different LUN. LUNs are complicated and difficult to get working correctly. I don't know how well the GVP card supports LUNs, either in hardware or software. At a quick glance, I don't see any jumper settings related to LUNs on the GVP 030 boards.
But LUN 0 on those card readers is the PCMCIA slot. It's sort of the default, fallback slot. Try using a CF card via a PCMCIA-to-CF adapter in that slot instead of putting something directly into the SD or CF slots. What happens if all slots are empty when you try to boot?
