It's a bit of a black art it seems.
I've got Worms running enlarged in a 800x600 window (can't try any higher because my monitor is only 1024x768) using the Direct3D filter and adjusting the sliders to move the image and strech it. Aside from OpenGL which does practically the same thing, all the other filters don't work properly as you described above.
When using the direct3D filter, the options under Display seem to have little effect.
