You need more RAM. No ifs ands or buts. The CV64/3D pulls four meg of fast RAM before you even leave the gate. I have one on my A4000. If it cannot get that RAM it is not going to perform.
OS3.9 is also a RAM hog (at least from an Amiga point of view)
RAM, more ram and once you have that more RAM again. You need to find at least 16 more meg somehow. Does the GVP board have ram sockets? With Zorro II you are going to be limited in what RAM cards you can use.
I am fortunate to have 112 meg of fast RAM aboard. It does make a difference.