I am not against drivers but it's better to have both options-- being able to go direct to hardware as well as driver or API interface. I would think you already know some of the great stuff written using direct to hardware method on old PCs and Amiga. And it's a superior interface to have the standardized hardware.
Superior ? Is it superior that any change in the hardware (evolution means change usually) will break pretty much every application coded using direct access ?
And I think you underestimate the complexity of hardware today: it's not just 3 registers you play with. It's far too complex to be accessed directly.
Last but not least, software was bypassed for a reason at that time: software wasn't complete and/or slowed everything down. As big as the OS are now (and yes, every OS, including OS4/MorphOS even though they are both less heavier than modern OS) the resources needed are negligeable compared to what's available, so it isn't a problem anymore.
Accessing hardware directly only would introduce problems today.
Again: welcome to 2010.