Partly because there is more data having to be pushed around by the same chip than with AGA but mostly because the bandwidth on ZII and IV is not man enough really.
When you add up computation delays from the 040 and the 060 to bandwidth delays across ZII/IV you get sluggish updates that are most noticable on games.
I had the same problem with PicassoIV.
Mind you while the ZII/ZIV is sluggish Voodoo 3 over Mediator does not fare much better.
Just notch it up to old age ;-)