Is there a way to make the CD-ROM drive non-bootable from the early startup menu (can't remember). Try that.
It sounds to me like the Amiga is looking for a disk in the CD-ROM drive to boot the Amiga, and only gives up after many, many tries. (but that still does not explain why it doesn't then try to boot from the hard drive).
Also see if there is a jumper on the GVP board that allows extra time for drives to spin-up upon turning the Amiga on. My A2091 has this jumper.
Actually, my Amiga usually needs a warm re-boot after turning it on. It just hangs with a black screen most times, but works fine after a CTRL-A-A. Been this way for years, and I think has something to do with the drives needing time to spin up before being recognized by the SCSI card.