X11 (the GUI part of Linux) is very slow. Plus, if you have bad graphic card drivers, that won't help either at all.
X11 in itself is not slow; that is a common misconception. I am a chip designer and use X11 all day. Raspi X11 is slow because of unaccelerated drivers.
