If you're busy anyway, why not make a clockport connection too somewhere? It's just a pity that you need at least a '030 for USB.
I hope you can stick to the Ethernet idea. I mean, it would be ubercool to have a nice webserver (without serial cable) on an A500. And besides, it would be alot easier for transfering software/games for WHDload etc.
BTW, I too prefer expandable memory above soldered chips, allthough 32MB would be plenty...