-Make sure no other files reside in DEVS:Monitors. Only Voodoo should be there.
-Make sure voodoo.card is in LIBS:Picasso96
-Make sure pci.library is in LIBS:
-run pciinfo from shell to make sure the voodoo card shows up. If not, re-seat the card and/or use a different PCI slot.
-As already mentioned, start the driver by double-clicking voodoo icon before starting P96mode.
-If all else fails, try disabling "loadmondrvs" from the startup-sequence, and uncomment the commands of the default routine to start the display driver. Try this LAST, only if none of the above fix it.
I copied that over from Google, mind
And yes, you need to create/edit the mode using P96mode.