Just try out New Mode, found no option to use P96.
You need to have P96 installed in order for the screenmodes to show up.

I don't know anything about the Vampire HDMI output, but I assume it uses some kind of Amiga screenmode for its display, correct? If you go into the regular Workbench Screenmode Prefs program, what shows up in the list?