The CPU is not connected directly to the RAM. Not on the motherboard and not on an accelerator.
They are connected to a memory controller and not the same memory controller either. The Amiga motherboard has a DRAM controller called Ramsey and the accelerator has its own memory controller.
Accesses to particular addresses will have different timing. Accesses to motherboard fast RAM will typically be slower. However just having memory on the motherboard will not affect overall memory speed UNLESS it is being used.
But it wont normally be used because AFAIK on the Amiga memory is not in a general pool. Address ranges have priorities and accelerator memory will have a higher priority.
