Did you solder it yourself?
I think OR the parallel port is broken, OR the four player adaptor OR you are using a partially compatible joystick.
try it both with different joysticks, and preferably a joystick from Atari (since it's their standard)
If it's still not working, try to use another 4 player adaptor (for instance, solder one yourself, it isn't hard at all)
If it's all not working, that particular pin of the parallel port is broken
It happens... :-( I've had something alike before.