osx didn't reinvented the wheel, they just took kernel bits and bobs and made their sauce ..
and their product is kind of popular..
OSX is a bad example. Apple needed a new OS and spent alot of money on it, without it they wouldn't have been able to jump to yet another CPU & would be struggling to compete.
Take a look at how they solved the problems:
Old apps are run in an emulator, we can use UAE for that.
New apps run in a completely different environment, this is where your problems start.
You could base your new environment on linux. To make it practical you'd have to drop any idea of keeping C: S: LIBS: etc. An Amiga lookalike window manager is doable (in fact I think it's been done), unfortunately it looks pretty dated.
Or you could start from AROS, but you've just gone round in a circle. It's hard to do memory protection in AOS because the way it was designed makes it hard. Your only option is to change the API, but that one change will ripple through everything. Even if you ever succeed you'll end up with another AOS alike system that will get ignored.
Your only real option is to make AROS compatible with AOS4.