Sounds like a duplex mismatch; possibly NWay isn't working properly.
I'd try with duplex forced on and off and see which one works better. There's a chance that currently the switch sets duplex to on while the Amigas switch it off. Forcing it on might reverse the problem, so the only solution would then be to force it off.
Obviously with ADSL connections, upload is always slower than download, but 20-25 KB can usually be expected with 3 Mbps (384 or so upstream).
Additionally you might want to try an increased RWIN parameter for internet downloads; that way more data can be on its way before an ACK is required (especially for higher latency connections).