While not marketed to home hobbyists, there are things out there. Look up some evaluation boards for microcontrollers, they have buttons, LEDs, a serial port or parallel port, headers and sometimes LCDs and keypads.
Jameco has some stuff at
Long url edited by admin which might be suitable, some of their Basic Stamp boards should be suitable for learning stuff and tinkering and are priced for most mortals to handle.
There was also a DIY game console that came up in some discussion some time ago.
http://www.xgamestation.com/about_gamestation.php Looks like fun.
I work at Atmel for my day job, and I know we have various AVR and ARM eval boards that could apply, but I don't know any prices or anything, as an evaluation board they may start stepping into expensive territory for a hobbyists's budget. There's also the FPslic, which is an AVR and FPGA combined into a single chip, but I've also seen an Atmel FPGA eval board with a socket for a discrete AVR from someone called Kanda (AVR chip not included though).
Microchip (PIC), Freescale (Coldfire, 68HC11 etc), and surely many companies have ARM or 8051 things. You might also think about a Xilinx eval board and look into some of the free microcontroller cores on opencores.org. Of course you might have to deal with higher prices or short-term licenses with some of these, especially for FPGA boards with HDL synthesizers or simulators.
