Your config is way off - monitor permitting you should get about max 120 Hz on 800x600@24bit (85 MHz pixelclock; 10 MHz more available with overclocking). I'm happy with my 100 Hz at about 63 MHz clock, and the monitor runs about 63 kHz, easy for a modern one.
