I accidentally downvoted a question (misclick, drank lots of coffee heh...) and then upvoted the same answer. I noticed I was award the critic badge for giving my first downvote.
Why is it wrong? You indeed cast a downvote.
Whether you revoke it later or not isn't relevant, it was indeed cast for at least a few minutes (that's how long the badge process takes to run).