Amiga joysticks are hot-swappable. Don't hot-swap anything in any other port though.
I thought the only danger with hot plugging mice or joysticks were the plugs which had metal surrounds on the end that could easily be shorted to a pin if not precisely aligned during insertion/removal.