Strange...then maybe it's VisualPrefs or MagicMenu playing up or some other tool or patch doing it. It used to happen to me a lot, until I stopped using either.
There was a patch on Aminet to prevent screen-sweap freezes too, but I don't know whether that actually makes things worse...