Basically, should I up vote if I think it is a good question/answer in the Late Answers and First Posts review queues?
I just want to go further than muru's answer and say, if you think it is an upvotable post, or you can edit it to be one yourself, then please do upvote it . I feel that I see far too many No Action Needed reviews on posts, answers particularly, that deserve an upvote (or a downvote, or indeed a flag, or that obviously need editing). It's very rare I click the No Action Needed button - almost always you can do something with the post IMHO. If it's a good answer (I know it's hard to verify) to an unanswered question then please don't let it get past you without an upvote.
If you would have upvoted otherwise, yes. Voting, up or down, is one of the actions that are sufficient to complete your review of that post.