Welcome! There's no reason your hardware setup shouldn't work with WHDLoad. You probably just need to configure the software differently. For example, use P96 or CyberGraphX instead of EGS for the video card. Also, make sure to take your TCP/IP stack offline before starting WHDLoad.
For the 3640, make sure you have the correct 68040.library in libs:. I also find the NOCACHE and NOAUTOVEC tooltypes helpful for troublesome WHDLoad games.