Either we live with the fact that some addressing modes crash the app (I take your word here that these exception can't be emulated. Personally I consider it possible though with some magic. Remember how tricky 68060 exception restart is? Yet it can be done), or we do JIT translation in the first place, where we handle these missing addressing modes by generating native ColdFire code for them.
It's only a couple address modes that can't be emulated through the illegal instruction trap. Since the program counter is modified before the illegal address mode is detected by the Coldfire the trap doesn't know what address the instruction that failed was at.
The mpy/div instructions cam't be emulated with an illegal instruction trap because they are legal instructions on the Coldfire... they just work different.
My proposed solution to this was to write a full 68K emulator that tracks the original hunks from the load file so that the emulator patches the code on disk whenever it encounters an instruction that can't be emulated with a trap. The patches are done with illegal instructions that can be trapped. Once the program has been run enough a patch file can be built and distributed. The Coldfire can then trap and emulate the instruction. To run it on a 680x0 you have to catch the exceptions and patch the instructions back to what they were... how difficult that is depends on what illegal instructions remain unused on the 68K/Coldfire. Basicly UAE with the ability to patch the file or build a patch file.
A full JIT emulator is definately possible and would be simpler than for another CPU family. What it wouldn't do that the trap method does is take advantage of new Coldfire CPUs that add more compatibility with 68K code. At least not without changes for each new CPU. As newer CPUs are release the trap method takes advantage automatically because less traps are generated and the code doesn't care. We also have the instruction trap emulation code now.
The good news about the address modes that can't be trapped is that they are slow so it's doubtfull a compiler will generate them and they rarely appear in assembly code. These instruction/address modes only appear on the 68020 if I remember right.
I think your solution to the speed/AttnFlags/new api issue is pretty close.
Untill better 060 emulation is available let the speed suffer. Not much choice in the matter.
Create a standard api to query CPU type for new apps.
Hope for the best on old software.
I suppose the speed/AttnFlags could be set through a prefs option so programs that don't use the math chip could select 68060 routines for C2P and other functions that might benefit.