Is the ball mouse an original Amiga one or is it going through an adaptor? Some adaptors can cause this type of phenomenon and can be picky about the mice that work reliably with them. Also, a basic thing but something that still catches a lot of people out is the surface - some optical mice have trouble on shiny surfaces, even shiny mouse mats. And some ball mice can also have trouble on shiny surfaces, especially if the ball's light and the rollers dirty.
