I doubt it's the drivers fault. I've seen that kind of behaviour before when there's bad memory on the gfx card. Doesn't say if you got 2 or 4MB on the CV64 but if you got 4Mb try removing the 4 socketed memory chips. Hopefully that'll help, if it does try get hold of some old S3 PCI gfx cards with socketed gfx memory to replace the bad ones you have.
BTW... what buster do you have? Had to upgrade to Buster11 to get a CV64 to work in a friends A3000.