Okay. Sounds like you haven’t setup the RTG with Picasso96 (P96) properly. To fix this you need delete the Screenmode.prefs file by booting up without startup-system and using Amiga DOS commands. Preferably copy it first then delete.
Reboot the system and it will bring the workbench back into native Amiga mode Screenmode.
Then try again.