Again, a kernel is not an OS. Linux as an OS is much more than just the kernel. But I agree: basically every problem Linux has is down to being stuck on crufty, monstrous legacy frameworks for its userland. The difference is, it didn't have to be that way. It was created from scratch as a hobby project and opened up to community involvement, not hacked into its present state as a guerilla update to a closed-source OS that was effectively abandoned twenty years ago.
It's the kernel plus some core drivers, commands and utilities. It isn't X-Windows or window managers or GUI toolkits or desktop environments. Those are extras that come bundled with distributions. You can install Linux with no GUI at all.
The decision to use X-Windows wasn't so strange back in 1991, since they were going for something Unix-like and it was an established standard, and it meant that they could run software on Unix servers remotely which was very cool then and is still very cool now. Remember Windows 3.1 came out in '92 and could do none of these things, and wasn't even fully multitasking. X Windows was one of the best GUI systems out there at the time.
The problem with X is not the speed or efficiency. The problem is the protocol transmits only visual, not semantic information. Or in other words, you don't say to the X server "I want a button here" you say "I want a grey box here with some shading and text on it, and tell me when the user clicks on it". Which means the appearance and behaviour is all the responsibility of the software at the other end of the connection. So if you are running some software natively, you can customise its look and feel, but then if you run something remotely it will run according to the settings on the remote machine. What you get is the Window Manager running locally (the X Server end), and the GUI toolkit running at the other end of the connection (the X Client end). So bye bye any hope of visual consistency, and you can forget accessibility (which nobody thought of until years later). X Windows doesn't know anything about what your on screen widgets
actually do.
There's another peculiarity of X Windows that is rather limiting, and that is the way you tell it that a window should have a border. It seems almost like a hack, as if Window Managers weren't invented until later, but what you do is you just open your window with the root window as the parent, and then the Window Manager gets informed and it steals it, reparents it and puts the titlebars and whatnot on it. (I know this from my experiments modifying Fluxbox.) This means your application has no control whatsoever over where your window goes. You just have faith that the window manager knows what to do with it. Which isn't usually a problem. But it means it's impossible to do "nested windows" in client areas like Windows NT has. Well, you can make them yourself, and there are toolkits with this capability, but there's no way to get the window manager to manage them.
X has instead different workspaces to keep your windows separate and grouped, so your WM typically just opens every new window on whatever workspace you're currently on. (There is a feature in the protocol to set a window's group ID but it doesn't have any defined purpose or consequences so nobody uses it.) This is a bit like the Amiga's method of using multiple screens, except in X you can't make your own workspace and put your own windows in it. You just get four* of them. And that's it.
*Well you can change that number in your settings. Fluxbox, incidentally, has away to put your windows in preferred places, which involves identifying your windows
by looking them up by title in a supplied list. Ugh.