You can use a pc monitor on a SGI if you get a 13W3 to VGA adapter. But your monitor should support sync on green as well.
Or you can use a serial console.
As for alternatives to IRIX, yes you can use Linux or BSD but as SGIs use none iso discs for the boot media you'll probably have to net boot it.
Personally I'd stick to IRIX.
Gary