Make sure you have the proper driver installed for the card, and your particular OS (win version, and 32 or 64 bit, respectively) from nVidia directly.
The fact it shows fine in VESA mode means it works and if not giving you options to tweak the resolutions upwards due to lack of proper drivers. I've never seen a "dead card" work fine in VESA mode and not when the proper drivers are installed. Grab the proper ones from nVidia directly, not from the card manufacturer - all cards should run just fine with the makers drivers and few, if any - resellers/rebadgers like XFX ever custom roll their own drivers.
Check out device manager in windows and see what it tells you.
