I'm away from my home base this week, so I don't have my schematics handy. But I believe the joystick directional signals go through a multiplexer chip before going to Paula. That way, both joysticks can be read by the same 4 bits on Paula. If I recall, it's a 74LS157.
