The main rule of thumb I would use is to ignore thew fact that you answered it, and use regular downvoting rules depending on how you downvote. In one sentence:

>If you hadn't answered it, would you downvote?