Quake on 680x0 is only really feasable on 68060. It's mostly the calculation overhead that kills it - VRAM speed is a secondary concern that has more impact once you move to PPC / GL.
I have a BPPC 040 25MHz and BVision. Under AGA it managed around 3-4 fps, peaking at 6. On the BVision it also managed about 3-4fps, peaking at 6. In this case the AGA speed / C2P time was not the limiting factor at all. It was already so slow that simply rendering the frames killed it.
To really enjoy (software only) quake on 680x0, use a CSMK3 060 and a graphics card (preferably on as fast a bus as possible). I've seem almost 20fps on such a setup.
GLQuake2 on 68060 is also rather playable, if you have a supported 3D card. GLQuake1 is actually slower than GLQuake2 largely because GLQuake1 was a hack whereas Quake2 was written from the ground up with GL support in mind.