I can't remember (getting old) whether the A4000 had any nvram like the A3000. If so, I wonder if whatever is stored in there got scrambled, or is not compatible with KS3.2 and may be causing the delay. I used a program (SetBatt for example) to clear out and re-set the contents to my liking on an A3000. Perhaps that might help.
I have recalled seeing elsewhere that not having a real floppy attached can cause delays with 3.2, but have not been able to test that myself yet.