Alright, let's crunch the numbers:
USB is a CPU-driven, not DMA driven, system. That means for every batch call, the CPU must be utilized.
For a single mouse, you will, at peak, utilize 12Mbps, roughly 1.5MB/sec. Now, the Amigas RAM speed is only 7MB/sec. So, for a single, solitairy mouse, you'd be tying up 40% of RAM access.... just imagine adding a keyboard.