Downvotes as a result of "spam" or "rude/abusive" flags being cast and therefore automatically done don't cost you rep.
Any manual downvote you do should cost you one rep point. Unless the question is deleted in a certain period of time (I believe you regain that rep).
(At least, as I understand the system)