So how about speeding up the video clock , putting 1GB of memory on the card and let Saga have whatever it needs?
Making the ram bus wider might help, but there may be issues with doing that (including cost).
I thought they were at the clock speed limit for the FPGA already.