Is the OS4 port of DosBox actually stable? The last time I tried it - admittedly many years ago - it was buggy to the point of being unusable. Don't know if it's been improved since then, but it sounds like you're experiencing the classic problem of buggy/broken software.
If it is trying to open an AGA screen, start by making sure NTSC, PAL, and other native modes are removed from Devs:Monitors. That should force the system to stick to RTG and will let you see if the machine is locking up, or if the program is trying to open a (native) screen that can't support what it's trying to display.
