Despite all the design problems with the XE, as I recall it the AGP port is not one of them.
The main hardware issues are:
1. DMA problems with onboard IDE
2. Possibly missing onboard sound chip
3. USB problems - manufacturing error used resistors instead of caps, or vice versa. Can be worked around using a hub
4. Using both memory slots is problematic unless very precise identical DIMMs are used
If I'm remembering correctly, the one thing with the AGP slot is that you have to choose between AGP and the adjacent PCI slot - they can't be used simultaneously. But that's a "feature" and not a bug, unlike the other issues.
Caveat: I also remember that the hardware DMA fix introduced another side effect, but I can't remember what. My board is not fixed. I'm using a Radeon 9xxx in the AGP slot.
Perhaps it's worth noting that the Radeon didn't play nice with my SCSI card. The original Radeon 7000 worked fine, but as soon as I tried a 9000-series the machine wouldn't POST. It took me years to track it down to the SCSI card. Replaced that with a different model and now everything's fine. The new SCSI card isn't detected by UBoot, so I can't boot from it, but it works fine once OS4 is loaded.