Great answer Olsen,
A nice, concise description of the issues at hand.
In the 80's and early '90s I used other 68K OS' for process control, and as those were designed to be modular, incorporating alternate graphics systems was a bit easier.
As you've pointed out AmigaOS was never designed with graphics upgrades in mind, but it could be altered to make that upgrade easier to support.