I'd agree with Thomas; the battery backed up clock is read once by the system at startup (if found), and from then the time of day counter is responsible for maintaining the system time after startup.
If the problem is that the time drift happens with the computer is powered off, then it's likely to be a battery backed up clock issue. I've written test/calibration procedures for that here:
http://amiga.serveftp.net/RTCrepair.html#OscillatorCalibrationIf the time drift is while the computer is powered on, then it could be a problem with the hardware timers. Use Amiga Test Kit to sanity check the CIA timers:
https://github.com/keirf/Amiga-StuffThere could be an issue with the 28.3MHz oscillator, as the time of day counter is derived from this on many Amiga models; though if the clock is drifting that much, you'd probably have seen video issues if the 28.3MHz reference was that far out.
