Oh sorry. A bit tired and didn't quite register the bit about it only working in VESA modes. Have you downloaded the drivers from Nvidia webpage. Windows generally just has a basic VESA driver and you need to get the driver from the GPU manufacturer.
edit
Try removing all traces of existing drivers before re-installing.