I was trying to find the correlation between downvoting and losing 1 rep for the downvote.
If I understood correctly downvoting a post with a score ≥ 0 always takes away 1 reputation from me.
But I couldn't see a consistent pattern while downvoting a post with a negative score. If I'm not mistaken sometimes I lose 1 rep after downvoting a post with a score of -1 and sometimes I don't.
I used to think the pattern is much clearer with posts with the score of -2 or lower. I never noticed any loss of reputation after downvoting one such post (though I cannot claim that I have checked meticulously every single time).
But to add to the confusion I noticed today I lost 1 reputation after downvoting one ranty (not-an-)answer posted by a new user laced with expletives (this one, deleted already). At the time of downvoting it already had a score of -4 (0 - 4).
Disclaimer: I flagged the answer as "rude or abusive" from the review queue long before downvoting it and at the time downvoting the flag had a "pending" status.
What are the exact cases where downvoting leads to reputation loss? (Or rather when it does not do that)