The main rule of thumb I would use is to ignore the fact that you answered it, and use regular downvoting rules depending on how you downvote. In one sentence:

>If you hadn't answered it, would you downvote?
   

<sub>This has been made community-wiki, so feel free to contribute with edits.</sub>