my direct experience about various electronic gears, says yes.
We are speaking about of not only CPUs but as already said, other tied components as well.
If you don't need a heater, cool down heat in electronics is one of the main problem and yes it slows down speed. (other than reduce the lifespan-stability as you said)