@alexh, Bugs?
I think this is the right way to figure the chip out. Feed the real Agnus input to an FPGA Agnus core and compare output then one will get a signal when there's a difference.
The hard part is how to use the generated debug data. Signal integrity, ground plane, EMI, jitter etc are some basic "oops" in these kind of projects.
