It's a commendable idea, but it doesn't work in practice.
I think it's unrealistic to expect someone to write drivers for all the existing amiga graphics cards. Writing graphics card drivers is quite tricky and requires special kind of skillset. There aren't many around who would be capable, not to mention willing.
This would basically lead to a situation that instead of two systems that support different set of hardware we'd have three. Instead of two different systems with slightly different quirks, we'd have three systems with different quirks.
New driver system would also mean that 3d drivers would need to be written from scratch.
I don't have an easy alternate solution to offer, unfortunately.