Sussed it, swapped the cable that had the plug to the other side of the adapter and connected it to the joystick 1/mouse port on the Amiga.
So you are still using the original joystick 2 port on the Amiga for the second joystick and then connecting a Joystick and USB mouse to the adapter board. The second port on the adapter board can't be used unless you get another cable with a plug and connect that side up.
Tried a couple of USB mice, one a no name brand all work. Talk about night and day, the old Amiga ball mouse and auto switcher will be getting put away in a box.
The speed that you can move the mouse around at is amazing. Will be trying the drivers the seller emailed to me next. 5 button's and a scroll wheel. Will post about it in a few.
Now have to decide where to mount it in the case :-/