I have a USB->serial adapter
If that was true, it was completely useless for your purpose, it would even damage your Amiga. The Amiga mouse port is not a serial port. It is an Atari-like game port. No serial hardware would work on this port.
And no, you do not have an USB->Amiga adapter. What people try to tell you is that you bought a fake. It is a PS/2->Amiga adapter with an USB connector soldered to the PS/2 end.
So what you need is an USB mouse which can also be used on a PS/2 port. Most of these mice come with a small green adapter. You just replace that green adapter by your Amiga adapter. You don't need that green adapter, but if the mouse came with such an adapter, it is an indication that the mouse would work on your Amiga adapter.
Pure USB mice won't work. You need a mouse which can "understand" PS/2.
