What resolution do you use exactly?
I can play Quake3 at 1024x768@32bits at high quality preset without any problem on my Mac mini with 32MB vram (demo four runs between 40 and 120fps depending on places).
In background i also had 2 1280x1024@32 screens (one for ambient and one for OWB). But it doesn't matter, since the memory management is per-screen.
[EDIT] But indeed, with texture details at max, it consumes quite more video ram, and it's too much.
Call that 10MB for your 2 background screens, leaving you with 22MB, assuming they weren't paged out. You then have 3MB per frame buffer for the game, so if you are using double buffering, that's 6MB. You probably have a 32-bit combined stencil/z-buffer (8-bit/24-bit), so there's another 3MB, although you might have a 16-bit Z-buffer only. We are down to 13MB of video memory free for textures, less if we assume that there are other allocations going on.
As you can see, it all adds up. The moment you go for the highest texture detail, you are probably getting to the point where stuff starts having to be switched in and out of VRAM. Not sure how MOS implementation of CGX does this, but I suspect it's not as quick as it could be.