I don´t know the Amiga APIs or libraries well enough to understand this concept 
How, in this "vision" is support for various buses and cards achieved in your mind? I would think that this would be delegated to a driver that know the chipset. The next problem is that this chipset can reside on a zorro bus, pci bus - or heaven forbid usb bus - LOL
The key to this, I think, is one of abstraction. This "new" graphics.library would itself be agnostic to the underlying hardware and thus need to sit on top of something lower level, in my mind. The additional functionality presented by existing rtg libraries would sit on top of the same layer and just expose the relevant features of it that the new graphics.library doesn't.
This lower layer would be responsible for detecting/managing the hardware bus and memory space, thus providing a consistent API for actual chip drivers, such that it doesn't matter if your graphics chip is attached by zorro or PCI. For the most part I see bus and memory management as different aspects of the same task - essentially how you map the device's memory/register space into the system memory map such that the chip driver knows where the hardware registers/memory are located as well as the graphics.library's BitMap allocator knowing where to get video memory from.
I think a lot more thought is required though
