The main rule of thumb I would use is to ignore the fact that you answered it, and use regular downvoting rules depending on how you downvote. In one sentence:

>If you hadn't answered it, would you downvote?

<sub>This has been made community-wiki for a reason, so feel free to contribute with edits.</sub>