The composite out on the Amiga 1200 accepts a lot of interference from the rest of the Amiga circuitry. Also 800x600 60Hz is not possible on the Amiga graphics chipset AFAIK, try NTSC 640x200 non-interlaced and see if that works right on the scandoubler. If you're using PAL 640x256 that might be why the monitor is having trouble maintaining a sync. Some systems don't like a 50 Hz display resolution.
