Oh? Then I simply misunderstood what you did. In that case, there's no problem of course. Just upload the modified souces along with your binary (in a separate archive, preferably) and make sure you mention the license.
Or an offer to provide the source to anyone who possesses the binary should do to comply with the GPL.
