OK, a few things...
Which power button? On the machine? Or the PSU itself?
If the latter, it ought not to power up the machine when switched off. If the former, it's not a switch just a toggle and the sudden reconnection of power will cause enough current to flow in the right places, for long enough, to be interpretted as the "power up" signal with many ATX PSUs.
You ough not to be connecting a live lead to the PSU anyway.. should be plugging the lead into the PSU and THEN plugging it into the wall socket/supply board. The socket should be OFF if it is one that's switched (crappy US ones typically are not, but in UK make sure the socket is off) only then should you turn on the socket.
To clarify, the order is:
1. Plug otherwise unconnected lead into PSU.
2. Plug lead into wall socket, supply board, etc with socket OFF if there is a switch.
3. Turn socket ON if there is a switch.
You risk a spike otherwise, I have several times (because I was lazy) tripped the ELCB in my house by not bothering to follow my own advice.. potentially I might have sent a reasonably large spike through my hardware each time. Your call...
