The second step is to configure your preferred mixing setup. Choose the Settings -> Mixing menu item. You should get the window shown in the attachment.
In this window, you need to choose your output mode, mixing rate, number of channels, panning etc.
Obviously, the more channels you choose, the more CPU is required. I had no problems with 8 channel 14-bit stereo at 28kHz (highest you can get in PAL mode) on the basic A1200 with added Fast RAM. Your 030 should be able to handle it no problem. I've had 32 channels on the 040 without any real issues.
The smoothing option uses interpolation when playing back samples at a rate lower than the mixing frequency. This produces a cleaner sound but the CPU load goes up dramatically with the number of channels. Unless you have a very fast 68060, forget it for realtime playback, use it when you want to try the Disk 16-bit mode.
As you increase the number of channels, OctaMED SS recomputes it's internal mixing tables to make sure that you don't clip the output. This causes it to get progressively quieter. You can compensate for this using the volume adjust slider, which boosts the output beyond what is theoretically the safe clipping limit. In practise, it's almost always a good idea to do so. In 16-channel mode, I've used volume adjusts of 350% without clipping.
The mix buffer size, is as the name suggests, size of the mixing buffer. Larger mix buffers decrease stutter when CPU load is high, but also increase latency and decrease responsiveness. The only time I've used higher mix buffers is when using direct-to-disk output.
If you click the panning button, you'll get a sub window with sliders for each of the channels. Panning is from -15 (extreme left) to +15 (extreme right). The "free panning" option limits the volume to avoid clipping issues again. I almost always have it turned off and rely on the overall volume adjust to get the maximum non-clipped playback volume.
The effects button opens another window with a few basic SFX that apply globally, such as stereo widening and echo. To be honest, I've not much use for those, since I route my audio through external gear anyway.
Hopefully this answers a few questions
