The A520 modulator gets my vote.
Nothing really was a pet hate hardware wise, I thought it was pretty good overall (I didn't have a big box Amiga). There are some things that I think would have been nice...
1) A500+ and A600 to use 14MHz 68000. A relatively simple improvement that would have encouraged more people to get these systems, and existing A500 owners to upgrade, which might have made Commodore some money (and increased the market via second hand Amigas). It would also have been good for 3D games as CPU speed had a direct impact on performance.
2) In addition the old A1000 should have been retained and kept the same specs as the premium consumer machine of the day (A1000+). The A2000 looked terrible.
3) A1200 IDE connector. Not-quite standard PCMCIA connector. Never-used FPU area. Tiny trapdoor. Lack of fast-ram SIMM slot.
4) Brick-like external PSUs with thick cables (A500, A1200).
5) CD32 should have been 1MB chip + 1MB fast. Never mind AGA missing out an 8-bit chunky mode, but that's an entire different argument (AGA deficiencies / dream Amiga chipset / etc).