You need to enable RTG support in WinUAE (to fake a graphics card), have the UAE monitor driver installed in Devs/Monitors and you need Picasso 96 installed - then you can select RTG screenmodes in up to 32bit colour.
You should also be able to select a 256 colour desktop using a native Amiga resolution (640 x 512 etc.) although RTG is faster and generally the way to go.
Without RTG support, OS 3.9 boots in a crappy 4 (8?) colour screenmode.
