Unfortunately the video input has to be in REC601 format (Y'CbCr) sometimes known as YUV.
Normal Amiga output is of course RGB.
There is a SLIM chance that the VP101 video encoder, used to generate composite, has it's REC 601 outputs enabled.
If it does, it should be possible to create a clip which clips over the VP101 and connects to the Voodoo feature connector and allows you to display Amiga native screens both in a window and full screen.
Some more research required, and as I dont have either an A1200 or a PCI board, or a Voodoo 3 I dont think I will do anymore.
It is VERY possible that with only a small investment of time you could make a DIRT cheap, medium quality (some data loss through 2x colour space conversions), scandoubler which may have the extra cool ability to be displayed in an RTG window.