Also:
The 68000 and the custom chips alternate in their accesses, even cycles being for one and odd cycles being for the other. This means the CPU can never take more than 50% of the available chip/slow-ram bus cycles.
Further, the 68k has the lowest priority in the system and will grant all bus access requests made by anything else whatever. The custom chips, on the other hand, can lock the CPU out entirely if they are busy, the CPU is forced to wait until it gets let back in (only on its appropriate half of the cycles, possibly wasting more time until that condition is met).
True "fast" ram is not directly accessible by the custom chips, the CPU has unfettered access and achieves full-speed. You can see why fast ram is nice to have.