think that it's not booting from hard disk because the install has removed some of the files used by setautoboot - e.g. the device ID file).
This doesn't "compute". If it can't find the harddrive it can't find the files either even if they are there.
Maybe you should try and check everything with HD-toolbox (run hdtoolbox from a shell with the device as an argument) and check if the boot-flag for the partition is there. Maybe you need to install a never version of the filesystem as well.
3.1 should have no problem running on a 500, and I cant see why the Dataflyer wouldn't work either.
I for one would try to reinstall everything from scratch with the 3.1 install-disk. If you have everything on backup I see no risk in doing that.
But you need to call the device for the harddrive-controller manually when you start up HD-Toolbox.
Is there a switch on the controller where you can turn autoboot on and off? I know that my old GVP-controller had that. That wouldn't explain why it works with the old install though...
Well, these are the things I can think of.