Depends on what's actually running. Any code exceeding the cache will get some - permanent - speed penalty in slower RAM. Swapping from slower to faster RAM would make that code ran faster - at the expense of having to copying the memory tiles before execution, so only heavily used code will profit from that (the larger the speed difference, the bigger the trade-off for swapping may be).
Also a bit academic since virtual memory doesn't work so well on AmigaOS.