As long as you don't hot swap these devices you are OK. The danger is in plugging a device in with power to the computer. This will easily blow a CIA chip (or one of the jelly bean chips).
As long as you don't plug the mouse/joystick in with power on, you can rest easy.
