Although there is only one physical DRAM interface, the current core uses it as a single random access memory. As long as the address(es) are in a separate physical bank inside the memory it is possible to effectively do several reads at the same time. So, for the display controller it can bust read memory at the same time as the CPU without impacting its performance.
A clever memory controller will automatically open as many banks as possible at the same time, so by careful address mapping of chip / fast regions we could speed things up.
/MikeJ