Kennys right. running VGA modes on the AGA chipset sucks up most of the bandwidth.
If you wish to use both reliably, your options are:
1) A scandoubler/flickerfixer. This will allow you to show 15kHz modes on a VGA monitor, freeing up bandwidth for the serial port (and everything else on the custom chip bus).
2) A graphics card. Depending on your needs/cash you can go for a blizzardvision or either a mediator or grexx (if you can find them). The latter allow various PCI devices to be used, including IO cards (I think).
A graphics card, even the BVision, makes a huge difference to general use but old custom chip banging hardware wont be able to use it.
Whichever you choose, a tower case is needed.
3) A buffered serial port expansion of some kind.
