Same reason they put the memory limit on Windows XP as of SP1: shitty programmers.
I think you meant to say "shitty third-party driver programmers." At the time Windows XP was released, it was rare to see more than 2 GB RAM in any 32-bit Windows system, let alone a consumer PC. By the time Service Pack 2 was released, large memory configuration were becoming more common. Many legacy device drivers don't know how to deal with addresses above 4 GB, so 32-bit Windows client SKUs artificially restrict physical memory to 4 GB.
When an add-in card with a large amount of memory is added to the system, the memory is mapped into the area below 4 GB. E.g. A video card with 1 GB RAM lives at physical addresses 3 GB - 4 GB. The system remaps system RAM at those addresses to 4 GB - 5 GB. Since Windows ignores RAM above 4 GB for the sake of down-level compatibility, the remapped RAM is not usable.
Unfortunately, users expect Microsoft to maintain compatibility with legacy third-party device drivers, regardless of how poorly written they are. Now, you can run 64-bit Windows and be done with it, assuming your hardware vendor provides 64-bit drivers. (If they don't, I'd wager their 32-bit drivers are among those that won't work in systems with more than 4 GB RAM.)