It really depends on what you want to do with it. Raspbian can be fairly neat once you clean out the bloat. On the other hand, I only use mine as an email server.
I suggest that instead of cleaning out the bloat, you start from
the installer instead of using a preinstalled image. This way you can just not install the whole graphical desktop category of packages. The only hassle I had was that I had to replace some of the included boot firmware blobs to ones that support model B to get the installer to boot.
I've been using one as a server with Arch Linux, which has always been a hassle to maintain, and I switched to Raspbian as soon as I found out that you can shrink it down in the installer.
