Do you have enough RAM for the AmigaSYS4aga?
CD32 with a SX-1 will try booting from HDD if there is no CD inserted (and the SX-1 itself is not disabled from the switch on the rear). I have a Workbench 3.1 installed into a compact flash- hard disk in my SX-1.
One thing I have noticed is that if there is no floppy disk drive connected, it will complain about missing DF0 when booting.
The connector block between CD32 and the SX-1 is also very often having a bad connection and this has crashed my system several times. Cleaning the connectors is definetively a good idea.