The Pi is a fantastic toy, and it has a ton of useful applications (I use mine for video surveillance), but as a desktop system it's absolutely horrific. Framebuffer drivers were cool back in '98, but trying to run a full-HD desktop with it is just painful.
the pi now has a hardware accelerated X server afaik. pi2 should be a lot more usable from the desktop than the pi1.
Not that I use mine for that - I use mine for Kodi and emulation (in a picade).