Some more info after much investigation. My conclusion is that the scart splitter I use is causing the issue. If I connect the Amiga through the splitter(quality splitter sold on amikit) the screen rolls for the first 15 minutes or so then gradually stabilizes. If I connect the amiga directly to scart on TV it syncs immediately. So the splitter is probably "eating up" the signal in some way.
The fact it stabilizes after time leads me to believe that switcher / splitter has a dodgy Capacitor in it. Would be interesing to see the inside of that splitter. It could still be the amiga thats 'warming up' with a similar cap related problem, which becomes noticeable when the signal passes through that splitter.