Ok, you'll all hate me for this being so simple, but not as much as I hate myself. Plugged in a CD32 controller. Now all is well.
This is - by far - the strangest Amiga problem and solution I have ever encountered. I hope I never deal with something like this again.
So, for the record, A2630 + lowlevel.library + no joypad = problem. Madness. Absolute madness.