The audio circuit needs +12 and -12V to work.
Reliable operation is in 10.5~12.1V range (symmetric to the negative line).
Distortion in audio indicate bad audio decoupling capacitors. Namely C433 and C443 in big-box Amigas. 22uF x 16V. Prefer swap 'em for no-polarized version. Using regular, polarized versions is a project error in the Amiga boards!
Also Paula can be involved in the audio problem, since she is the responsible for sound generation (among other things).