Well, I get concerned about swapping out frequently just from physical wear and tear.
I have a couple of those switch box things you plug two joystick into, and that makes me feel more comfortable. One is going bad but thats easier to replace than if you accidentally damage the port on the computer itself