A lot of it has to do with chip-RAM throughput. The 6-plane 320px modes are pushing the limits of the bandwidth alotted to the bitplane-fetch channels. 640px mode requires twice as much data for the same bit-depth, so 6-plane 640px is well beyond its capabilities. (IIRC, 4-plane 640px takes all the video bandwidth on OCS.)
As for 640px being slow, that's because when running from chip RAM, the CPU has to compete with the chipset for RAM access, and the more the video takes up, the less chance the CPU has to fetch instructions and data, so it winds up just spinning its wheels more often than in less intensive modes. (This isn't a problem when you have some fast RAM, of course, but since none of the OCS home-computer models shipped with fast RAM, and the A500's default 512KB expander is "slow RAM" that's subject to the same restrictions as chip RAM, it's understandable why programmers shied away from it.)