As for me, 1920x1200x24bits turned out to consume more than 50 of my 64MB of video memory, so I changed the screen's deepness to 15/16bits until I manage to flash a 256MB card 
I would guess that you are running out of VRAM and need to lower your color depth as suggested, or find a smaller monitor to use with your MacMini.