It really depends on what you want to do with it. Raspbian can be fairly neat once you clean out the bloat. On the other hand, I only use mine as an email server.
I'm using one of those 3rd party minimal Raspbian installations and I'm building my server over it. Too bad Raspberrys can be unreliable with SD cards, so I wouldn't use it for lots of writes and too important stuff... unless you have external HD to store stuff, it might be better.
I planned to use my 16GB class 10 card for server, but it has got corrupted few times already. Smaller class 2 and 4 cards seem to be more reliable... so I ordered 32GB class 4 card and will see if it would work better.