You must have done something wrong. The increase of image decoding speed (specially of JPGs) should have been easily noticable on a 604 running at 233Mhz, without the need for a stop watch.
Image decoding is still much faster on a slow 603 Vs a 68040. I haven't done any comparisons with a 68060, but many people have said that the 603 still outperforms it.
Sorry, but i forgot an important factor regarding this issue.
Were you running AOS 3.9 when you did the tests? If you didn't, then the OS was using a 68k-only picture.datatype, which would explain why you did not experience better performance due to the excessive context switching, as you said. What i wrote is true for AOS 3.9 where the picture.datatype is 68/PPC.