Most other OSs have memory protection for process protection at least.
Sure, but that affects normal programs too. JIT compiled code doesn't normally run under a further level of protection, which is what your statement implied.
Self modifying code is a performance limitation on real processors also. A real CPU can have hardware help to snoop out the writes but flushing and reloading the ICache still has overhead, especially if the code has been fetched by the CPU. I suppose the JIT could use the MMU but often changing the way pages are marked and interrupts from responding to page violations may be more overhead than it is worth.
That is why I suggested implementing the code cache so that it mirrors the 68060 instruction cache. Code that works on a 68060 would automatically work on the JIT. You don't have to snoop as the 68060 didn't, just flush the JIT cache when the program clears the cache. The reason to implement it with the same size as the 68060 is in case something is modifying code that would have been automatically retired from the 68060 cache.
Code that fits in the cache would run fast in the JIT, the same as the 68060. Code that doesn't fit in the cache would run slow in the JIT, the same as the 68060.
The problem comes when you want to accelerate code that was slow to begin with, because this is likely to slow down code that was fast to begin with.