I think the PET user port is not the same as the VIC-20 and C64/128's user ports.
The VIC-20 and C64/128 user ports aren't the same. There are differences in the 1-12 pins, only A-N are the same. As long as you avoid the differences then it's possible to create a device that is compatible with both.
The PET user port is also the same on the A-N pins, so it's certainly possible to make a device compatible across all three. Except you don't have any 5v/9v power, so you would need to cut those traces on the adapter and grab power from somewhere else.
http://www.hardwarebook.info/Commodore_PET_Parallel_User_Porthttp://www.hardwarebook.info/VIC-20_User_I/Ohttp://www.hardwarebook.info/C64/128_User_PortThe plus 4 is entirely different, as A-N are no longer connected to a CIA/VIA. However they are connected to the 6551 UART, so RS232 (and only RS232) devices should be able to work across all four. The power pins are there too, so no cutting traces. However you will need to make sure that you are using the correct polarity for all the pins, as you can't do cheap hacks on the plus 4 by violating the RS232 spec.
http://www.hardwarebook.info/Plus4_User_Port