I am not sure what revision(s) of A3640 were included with A4000T. Probably 3.1, as they were the most commonly produced revision.
Try setting up an IDE hard drive and booting into OS3.1 or better yet to 3.9 from it, and then start adding more stuff to your system, like the SCSI hard drive, etc. That would enable you to diagnose the problem(s) more easily.
Have you tried a different amiga mouse, or the same mouse with a different Amiga?
Finally, Vesalia seem to stock some A4000T spare parts, so maybe you can find the right replacement stuff there?
EDIT: Also, your A3640 card may be defective. I've heard that some revisions are prone to failures. Again, just a guess.
