Drivers conforming to which API, though? Consider windows alone. What you commonly consider to be the "driver" for a given graphics card usually ends up being a range of drivers that allow it to work with various APIs (basic 2D, video, 3D etc). You've then got different versions of those drivers for different versions of the API's they provide the services for.
If you look at the DirectX drivers for a typical modern graphics card alone, they can be several tens of MB in size and that's before you throw in support for DirectX 7,8,9,10 interfaces that could reasonably be expected to exist.
OSX also has multiple driver layers to support various APIs. There might be a single unified driver at some layer which provides all the services required by Quartz, Core Graphics etc. but even then I doubt it, to be honest.
Providing a hardware level detection and capability reporting really is the most pragmatic choice and funnily enough, that's generally what the cards have already.
