for me it's fuzzball, no doubt about it.
it has about 50 levels or so, i never made it past level 8... and i tried sooooo many times.
but beeing 'most annoying' is only half of the truth, the other half is, that it is highly addictive.
if i play it, i shout, i scream, i beat up parts of my furniture, i bite into the joystick, but i cant stop starting over and over again for hours ;-)