My A3k has been running a Merlin in Z III mode for 14+ years. No problems apart from some glitches the Merlin has.
A Buster 7 works AOK most of the time, at least if there's only one Z3 card active. What it simply
can't provide is Z3 DMA - you'd need a Buster 11 for that (for a single DMA-able card), but from the Deneb FAQ I've learned that an 11 doesn't even help with a stock A3000 since the onboard SCSI already counts as Z3 DMA. So you'd need SCSI on your accelerator card to run one Z3 DMA card.
Further reading:
http://www.thule.no/haynie/chips/buster/docs/buster2.pdfHowever, RTG cards are non-DMA so it's a completely different story. What board rev is yours? There might be heat and timing related problems with some board and PAL revs. If you have PAL programming facilities at hand, some of the PAL codes are available for testing:
http://www.thule.no/haynie/systems/amiga3k/pals/