The main rule of thumb I would use is to ignore the fact that you answered it, and use regular downvoting rules depending on how you downvote. In one sentence:
If you hadn't answered it, would you downvote?
This has been made community-wiki for a reason, so feel free to contribute with edits.