That's not strictly true, downix - you can have both processors running code from cache and only halt one to fetch more instructions for the other. The amiga RC5 client works like this, cracking with both PPC and 68k simultaneously.
For all other stuff you're right though. They have to be swapped constantly, flushing the cache each time. Not good.