This could also be part of your problem. Frankly mp3 decoding can be heavy work for a 680x0. There were integer versions of this library that are supposed to be a lot faster if you don't have a 68060.
on every version of mpega.library (fpu or integer) mp3 in high quality cannot be played with 68040

it can play at high quality only in mono...
On Amiga howerver you can install Delfina audio card, its DSP processor can play mp3 with no cpu load

With Soundblaster 128 we can get the best output quality.