It sounds like your BENQ is not compatible with all the display modes the Amiga can output. Most likely it can't sync down to 15KHz (this is the most common issue people face when trying to use modern display devices).
I'm not sure what you mean by hooking up a Mac Mini via RS-232. That's a serial port connection. I assume you mean the HD15 VGA connector? (it's blue, has 15 pins vs the 9 pins the serial port has).
In any case, if you want to do this with minimal hassle, pay the money and get yourself an Indivision. Looks like they're running around $140 USD on AmigaKit right now, other vendors may vary.
http://amigakit.leamancomputing.com/catalog/product_info.php?cPath=82&products_id=1148There's lots of other solutions. How much is your time worth to you to try to kludge one of them into working? :lol: