After a long looong time of troubleshooting and pulling my hair, then I did figure out the issue for this one - it was missing GROUND.
I did dismantle my keyboard, and did have the print "hanging loose" - not screwed in place, as I was doing test. I did have this setup for the most of the time, until I gave up for now and screwed everything in place again, to not loose the screws - and then it suddenly did work!?!!
After some retesting, I could confirm this - when the keyboard print is not connected to the ground plane of the keyboard, then you will experience this problem - or rather, at least I did - not sure if this is typical or not.
Quite a weird problem for me, but luckily it is now solved and partly understood 😂