RTGMaster: No appropriate screen modes available!

[Piru]

"RTGMaster" is probably one of the worst piece of sh*t software ever written for Amiga. I doubt you can get it running in any good way. Good luck, though.

[Piru]

is there any alternative if the software requires RTGMaster to display on a CGX screen?

Well, I'm afraid the only option is to try twiddle with the options until it works. That is, if you get it to work at all.

Ilkka Lehtoranta has written a MorphOS rtgmaster reimplementation but it obviously won't be much of help for non-MOSers. It does have relatively good built-in CGX support, however.