Downvotes as a result of "spam" or "rude/abusive" flags being cast and therefore automatically done don't cost you rep.

Any manual downvote you do should cost you one rep point.  Unless the question is deleted in a certain period of time (I believe you regain that rep).

(At least, as I understand the system)