The main rule of thumb I would use is to ignore the fact that you answered it, and use regular downvoting rules depending on how you downvote. In one sentence:
If you hadn't answered it, would you downvote?
Especially, you shouldn't downvote early to get your answer up for the OP and others to vote on more, and you shouldn't do this just out of jealousy for another answer. In my opinion, these are strict no-no's.
This has been made community-wiki, so feel free to contribute with edits.