Correct me if I'm wrong, but isn't Altivec just a fancy name for the maths processor in the MPU? If so, then Altivec is to PPC what the 68881 FPU was to the 68000.
Any program can use the Altivec registers to its own advantage, but only if it's the only game in town. As soon as other tasks start using the registers as well, the exec needs to save and restore them each time there is a task switch. More overheads in the task switching process.
tony