The overhead is tiny. In MorphOS all old AmigaOS API calls are called as 68k from PPC code and the overhead is negligible. Overhead is so small that even OS4 cant beat it with interfaces.
SetFunction() is trivial. In MorphOS you can patch 68k functions by PPC or 68k code. You have to include a 68k gate to install PPC native patch but internally 68k code is never executed in PPC -> PPC calls. PPC native patches run without performance loss and on the other hand 68k SnoopDos is fully functional in MorphOS if you dont happen to have PPC native port on your HD.
Not being source compatible doesnt matter. You cant port MCP or MCX or SnoopDos from 68k to PPC or x86 or ARM without making changes to patch code. In 68k arguments are passed in 68k registers and they often involve hand written machine code.
It's got to execute the 68k gate, because it doesn't know what the code is doing. You can dynamically recompile it, but you've still got some lookups to do to find the native code. Or you can put some heuristics in to find a standard gate and shortcut it. It's not just old libraries that this would have to happen for if you want to be able to drop ppc libraries in and have them used by 68k software. It could be any library that has ever been written.
You can write libraries completely in C, I wouldn't like to say how often in occurred but you don't need assembler at all (just compiler support for regparm).
The next problem is that we're actually talking about x86 not PPC. So all of your code is going to have to do endian conversions. You either go the amithlon route, but then software that needs to write little endian values will end up doing two conversions. Or you spend the rest of time changing code to do the conversion where necessary. Or you try to come up with a new compiler patch that allows you to selectively have little or big endian variables.
Because of the amount of effort, still being limited to single processor and no memory protection, how horrible all the compromises would be to get it working on x86 it's not going to happen. I can't see developers flocking to an operating system that is crippled with that amount of backward compatibility. You can't even support x64.
ARM is probably a better choice if you want to avoid the endian issues, but then you have new problems like lack of decent hardware. Rasberry Pi is cheap, but it's also not very good. Even the more expensive boards are pretty poor in terms of expandability. 64bit is also coming to ARM, but you'll be shutting yourself off from that too.