Oops, didn't read all of your post.
If I copy the library then setpatch runs and the system reboots, then the it hangs on setpatch when the system next starts up.
I'd suggest to boot from the Install3.1 disk (or Install3.0, whichever you have). It has the original 68040.library on it and should work. Then install OS3.1 on the HDD and copy the latest 68040.library to it. Boot from HDD. Now create an OS3.9 emergency disk. Check that the 68040.library has been copied to the disk. Now boot from the emergency disk.
If everything up to here worked, you should be able to install OS3.9 from scratch. The 68040.library from the emergency disk should automatically be installed, too.
If you have still problems, even with 3.1, then it's probably a hardware problem because it only causes problems, when the 68040 is running at full speed. Perhaps a faulty RAM chip.
Bye,
Thomas