Many games only use the game port (port 2) so you wouldn't need to keep unplugging the mouse all the time. The first port is usually only used for joysticks in two player games and such.
For the mouse issue... is it possible you have an Amiga/Atari mouse with the switch in the Atari position? That would cause all movement to be very glitchy.
