I bought Geos for the C64. I liked it. I do not know how well it went. I think it was fairly popular. http://www.bombjack.org/commodore could show how popular it was by looking at how many magazines had content for it.
I used Geos and Jane on my C128 for a good couple of years before I got my first Amiga.
There are probably lots of OS that were unsuccessful that we will never know of (ipso facto)
I'd have to say MSDOS 4 which I think was more damned than Windows ME as an OS.
MS DOS 4 was so buggy on many machines it was un-usable. I remember people went back to DOS 3.3 in droves.