Downvotes as a result of "spam" or "rude/abusive" flags being cast and therefore automatically done don't cost you rep.

Downvotes on questions cost nothing.

Any answer downvote you do should cost you one rep point.  Unless the question is deleted in a certain period of time (I believe you regain that rep).  (This excludes flags triggering a downvote)

(At least, as I understand the system)