If you're using WinUAE to run OctaMED SoundStudio, try using the MaestroPro 16-bit sound emulation of
TheMaestrix in conjunction with the AHI driver that came with WinUAE. It will require a full install of
AHI to do it though.
Once you've got the software installed and the Font issue resolved, this should give you the best quality sound you can get under WinUAE. Also, if you need a link to OctaMED SoundStudio, look
here.