I'll have to call shenanigans on this. I experienced measurably better performance out of particular serial drivers. Without looking at my old Miami config file I cannot recall off the top of my head which one, but I was able to get stable and reliable performance with a 56k modem on one of the alternate serial.device drivers whereas I could not with others.
Lucky you
Back in the day I toyed with the idea of rewriting the serial.device driver in plain 'C' just to prove that it can be done. As part of this exercise, I reviewed the original serial.device source code which, strangely enough, wasn't poorly written at all.
If you properly did your job in the network front-end (slip.device, cslip.device, ppp.device) which called upon the serial driver to move its data around, and enabled the recommended control options (8N1, RTS/CTS, high-speed mode) you enabled a code path through the serial.device RBF interrupt handler which was as short as it could be. From what I know, your typical custom serial.device variant reproduced exactly this code path, and removed every other optional code path which the original serial.device had to support (7E1, xON/xOFF, etc.).
The big problem remained, no matter what device driver you put your trust in: the receive buffer of the built-in serial hardware has just enough room for one word to be stored at a time. Once the receive buffer is full, the hardware triggers an interrupt and your interrupt handler has to be fast and lucky to pick it up before the next incoming word overwrites it. This works reasonably well for transmission speeds up to 57.600 bps (assuming fair weather), but gets flakier the more you demand of the hardware.
If your display mode (e.g. 16 colour hires interlaced) puts extra load on the bus, you'll make it harder for the rbf interrupt to be serviced quickly. If you write to disk, hit the keyboard, etc. you're asking more interrupts to be serviced while the serial rbf interrupt also has to have its day. If you run more programs in the background, same effect.
It's easy to throw off the serial receiver because it's so easy to trip it up at high speeds: incoming words are getting more likely to be trashed because there is basically no safety buffer to avoid exactly that. There is very, very little which your serial.device driver replica can do to alleviate the situation.
What works consistently well, and very well indeed is to use serial device hardware which has a FIFO, and which can be found on any Amiga serial card made since the early days.
