Dave: do you hear some high pitch when trying to move the mouse? If so, you have a (near) dead Paula.
Other than that: can be the connector, one of the CIAs, a corroded track, you name it.
I looked and then realized that the A4000 does not appear to have the pair of CIA's like the A500 & A2000 that I have replaced before to correct problems with the mouse. I don't hear a whine, but forgot to mention that the mouse does move left to right, but not up and down the screen and I have tried several known good mice. Everything on this A4000D motherboard is surface mounted anyway, so I can't fix it myself.
Edit: My 15 year old Son reminds me that I am getting older and that old people cannot hear certain frequencies every time I turn off any Amiga I have connected to an old NEC 3D (think that is the model) Multisync monitor that is still turned on and it emits a high pitch noise that drives him to cover his ears and I cannot hear the noise it makes at all.