I use a Cocolino adapter, with an optical Logitech wheel mouse. It works flawlessly. I never noticed any slow down, but just ran SysSpeed, and I see what you mean; it does indeed steal some cpu time. This is on my A1200 for what that's worth.
I doubt I'll let this bother me.
