Interesting, I seem to remember having a similar issue with a 2630 and a GVP HC+8 card. I think I ended up disabling the RAM on the GVP card and sticking a SupraRAM card in set to 4MB. (I was still using the SCSI on the GVP card.)
Incompatibility? Maybe. It was 2 years ago and I no longer have the hardware to play with.
Slightly off topic, but if you're in the market for a different accelerator, I've got a GVP '030 with 4MB onboard I've been too lazy to list on eBay...