I guess it all depends on what you plan to do with the machine. If it's for nostalgia, circa 1991, then by all means leave it stock. Pretty much all the games on Amiga, except a few of the later titles, were designed to run on a stock machine.
I have had a LOT of compatibility problems with my DKB Cobra and older games. Unfortunately for me, the Cobra is about as easy to put in the Amiga as trying to push a thumbtack in your eyeball. At first I had removed all the casing from the amiga so I could rip out the Cobra easily, but that ain't good for the machine, and constantly removing the board wasn't exactly good for the connector either. Without the accelerator though, I have almost no compatibility problems. Some of the compatibility problems can be solved via the early boot menu, or with the use of tools like Degrader.
*BUT*, without my accelerator (and the 32mb of ram with it) using the workbench is...something else. You have to keep it as feature-less as possible to preserve RAM and performance and that ain't fun at all if you plan on using the amiga for more than games.
If you can afford it, get a 1200 and accelerate it so you can enjoy the latter part of the Amiga software history. Then get an A500+ or an A600 (most of them are dirt cheap now) and keep it away from compatibility-impairing hardware such as accelerators. HD's and RAM expansions are nothing to worry about. Personnally I'd go for the A600 if you're a little short on cash as the 500's HDs are getting rarer, and more expensive whereas you can just shove (almost) any 2.5" IDE drive in a 600 assuming it has KS2.05 (prior versions did not have support for HD or PCMCIA).
But don't expect PC-like tweaking. On a PC you can get anywhere from 10% to 500% speed increases with only a couple of BIOS settings and a cooling system. It's quite the opposite on the Amiga...Don't forget that while you can accelerate the processing power of the machine, most of the hardware inside remains that old 198x/199x technology.