I use Linux on x86, PPC-Mac and recently on my SEGA Dreamcast. Xwindows does use a lot of memory, but there is a very good alternative - TinyX. Its designed with embedded systems in mind with few resources, but will work on any machine capable of running the linux kernel.
Taking the embedded linux approach should produce a fast and stable linux distro, without losing out too much on the modern looks.
