Ah, so the plungers have a conductive coating on them. That would explain why this one didn't work, it's worn off.
I checked some A500 keyboard repair videos before looking inside the keyboard, and only warning I got was not to use IPA on the membrane. Luckily I only rubbed a bit on the tip/edge of each side (the parts that touch the membrane), not the whole thing of the broken plunger, which was already not working anyway.
Still wonder why that one and not any of the others is so worn out, and the color is faded, while all the others are bright and nice and work.
So, need to get a new plunger then I guess (won't go through the trouble of recoating it myself), if I want the other key to work as well. But, ithat one is not that important anyway.
The guy I bought the Amiga from said everything were tested and working before it was shipped to me, even every key. So guess he forgot to check that key.
Edit: Ordered 3 new plastic bits with plungers on them from ebay. One for the broken key, one for F3 which works but have to press really hard, and one more in case I might need it in the future.