AFAIK Toni uses Vista for developping WinUAE, so it must run at least for him. And I can say that it runs flawlessly for me, too, both on Vista RC1 32bit and Home Premium 64bit on an AMD Athlon64X2.
The rtg.library message has nothing to do with Vista, it appears on every Windows version. It means just what it says: you have installed an old version of rtg.library on your Workbench. This probably also causes the gfx glitches. Copy the new one which comes with WinUAE to the appropiate directory and you'll be fine.
Bye,
Thomas