Sounds like the memory might be overheating as I've had similar display corruption issues when overclocking the memory on nVidia gfx cards.
It looks like it's passively cooled so one thing you could do is screw a little fan onto the heatsink. Or just make sure that you've got plenty of air flowing through the case.
Either way I'd RMA it and see if another one is any better. If not then try one of my suggestions.
Andy
