SPI is too slow, you need something that can handle about 15MBytes per second for basic use and about 30MBytes for ZIII to work. You can't cut it too close, you need some headroom to make up for any small delays that might occur.
Latency is another issue. If the latencies are too high, no sustained transfer rate is going to help you. You'll stall the bus and at best have retries and wait states slowing you down.
I've done a TON of research on this same idea in the last few years (and still am, but not as heavily). One of the reasons I keep putting it off is the fear of rejection by users. It's not a "real" Amiga, it's just an emulator, etc. You have to admit, we're an easily offended and fickle group.
The other is getting a fast link between the CPU and the Amiga bus. Most SOC's have limited IO capability. GPIO isn't fast enough on any I've seen and local buses are long gone. PCI-e is on some of them, but that's not a cheap interface.
The other big stumbling block is ZorroIII. IMHO, forget Z3, it's not worth it for the 4-5 available cards that are easily replaced with faster components on an SOC. Z3 never worked right anyway, you get at most one busmaster out of the 4 available.
Edit: let me clarify on the Z3 issue. If you don't allow Z3, you only need to communicate with the system on the lower 16MB, using 24 bit addresses. This would apply to any Amiga model. Everything above that could be local to the SOC.