I too have experienced similar problems with my CV64-3D with scandoubler on an LCD monitor. I can get 1024x768 and 800x600 to work but nothing else. AGA screen modes scandoubled are displayed but with a "frequency out of range" error.
What I've found is that the range of supported modes on LCD screens is not as wide as traditional CRT's. I have tried 15" and 17" LCD's and they both have issues.
What I have done now is to use 1024x768 for workbench (which works) and for other modes not supported by the LCD I use a CRT hooked up as a second monitor. Not an ideal solution but it works.
Fiddling around with P96 settings, try turning interlace on, scan double off, or other combinations can help.
Weed
Weed