Ok I finally done it, here are some of the tested frequencies using a Buster 9 (if it has anything to do with it anyway..).
For some modes I lowered the Sync Lengths the lowest possible to be able to raise the frequency a bit. I don't know if all monitors will accept these. Values presented are the best I could get for these modes but I can't remember if I tryed to lower the sync legnths when the attained frequency was already high. I only present the synch lengths for the 1024x768 and 800x600 modes, both at 24bit.
Pixel clock also had to be adjusted cause above certain levels image gets messed up. Pulse offsets didn't make any difference so I kept them under normal values (just clone an existing mode and use it's values)
800x600 24bit:
Scan Mode- Normal
Pixel Clock- 55
Sync Lenghts:
Horizontal- 232(doesn't allow less)
Vertical- 47 (maybe less is possible to raise frequency didn't try)
Frequency- 82,37Hz :-)
NOTE: WHEN THINGS MOVE IN THE SCREEN I GET THOSE ARTIFACTS SOME OF YOU MENTIONED BUT NONMOVING PICTURES ARE PERFECT.
TO TAKE THEM OFF I HAD TO lower the pixel clock to 32 wich resulted in 49,85Hz (You might want to try to lower the vertical sync length)
1024x768 24bit:
Same values but lowered the synch lengths to maximum: 232(doesn't allow less) and 12 (less would make the picture disappear), for horizontal and verticall ones respectivelly.
Frequency- 56Hz
16bit Modes:
NOTE: I didn't had any of the mentioned artifacts that I got when moving things around the screen with 24bit.
1024x768
Other values as previous mode. Pixel clocks maximum value is 107 otherwise picture gets messed up.
This resulted in 88.54Hz.
@1152x864:
Other values as previous mode:
Frequency- 72,4Hz
1280x1024
..
- 57Hz
1400x1100
-49hz