That depends how you look at it.
If the hypothetical DSP-A3000 (what was the model name for it again?) was released with a strictly functional(help me out here, that is not exactly what I mean to say) API on the 68K side with no access to the DSP "behind the scenes" then the upper limit would be whatever C= implemented. They could have added 24-bit functionality with a disclaimer that "it might be scaled down to 16 bits".
If the API had measures for uploading code as well then the DAC might or might not be set in stone. Again depending on what you would be allowed to do with the DSP-internal API.
As for sound quality that is always hard in an electrically noisy environment like a motherboard. One could envision other models with a pure digital out that you could convert to analog outside the box, but then you'd lose the chipset sound.
D/A converter quality will always affect the end result. That is no different to how it is today in both computers and hifi equipment.
If you want an approximation then I believe several Amiga models can connect the sound out of a CD-ROM to the motherboard and have the sound mixed out the phono plugs at the back. That should be a pretty good indication of what you could have expected.
As for the capabilities of the DSP chip itself I'm unsure, but I think it is probably stronger than the Motorola 56000(?). That should have room for today's advanced sound decompression, sample mixing, volume scaling, echo, delay, reverb etc, plus realtime sound generator. All that is enough to wish for some C= directors head on a stake. As has been seen on the Atari Falcon a dsp can also be used for other mathematical purposes.
As for how good it would have been: I imagine there would have been a lot of noise from "hardcore" game developers as you would probably need to keep the OS alive to make it function. They would have accepted after a while I bet...
A DSP is still a garbage in-garbage out device, but seeing what people have done with the limits of 4 8-bit channels then you can only wonder where we would have been.
All good. In My Opinion. But you might want to rephrase your question?
