Ok, we've established that the clockport Delfina is almost useless for my original idea since it doesn't have the bandwidth to support multiple streams at 44kHz.
However, it looks like it could manage around 5 - 6 mono 22kHz 16-bit source streams before total bus saturation which means it might be able to serve as a cut down sampler. There are also various lossy 8-bit delta encodings one could apply to the source data increase that again, giving a more respectible polyphony.
The second idea I had might still be ok, which was to investigte the possiblility of a tiny softsynth that runs entirely on the DSP (it has several useful lookup tables in ROM) and simply controlled by signals from the CPU that are ultimately raised by MIDI input. The clockport has more than ample bandwidth for this.