5

I encountered this review audit. To test me, the review item told me the question had received close votes as primarily opinion-based. This was obviously wrong, which I recognized quickly, but I saw a comment pointing to another question. It seemed like it might be a post might be a duplicate, so I opened the post in a separate tab, which revealed to me that the it was already closed as a duplicate--though of a different question than suggested in the comment.

(The question says it was closed as a duplicate by the Community ♦ bot, in case that's relevant. Since Community was listed as the sole close voter, I believe that simply means the question was closed after the OP confirmed to the system that they agreed with its duplicate status.)

At this point--since it was a Close Votes review item for a question that was already closed--I knew it was a review audit. I suspected this might be a new kind of review audit, where a highly upvoted but correctly closed duplicate would be used to test reviewers ability to select a reasonable close reason. I was pleased and intrigued at this possibility.

It also occurred to me that this might be a bug in the system, such that voting to close it as a duplicate of the question it's already duped to would make me fail the audit. But I figured there was only one (reasonable) way to find out, so I proceeded, and voted to close the question as a duplicate of what's already duped to. That is, in addition to correctly voting to close it as a duplicate, I also selected the correct duplicate.

Then the system told me I had failed the audit.

I believe this particular review-auditing behavior is both unintended and undesirable, and furthermore that it is not the result of errors made by Ask Ubuntu users in incorrectly closing questions (like some bad audits are) but instead is a design flaw in the system.

Either that, or I am missing something I should be noticing.

Note that this is not the result of a race condition: the post used for the audit was not closed between when the review item loaded and when I made my decision, but some days previously. (It was closed on March 30th and I reviewed it April 8th.)

6
  • A growing problem made by many others
    – Virusboy
    Commented Apr 9, 2015 at 0:34
  • Now that doesn't seem right.
    – Seth Mod
    Commented Apr 9, 2015 at 0:37
  • Re: race condition, but when was the question selected for Auditing? (a la this audit I failed)?
    – muru
    Commented Apr 9, 2015 at 3:16
  • 1
    @muru That's a good point--though in this case that would mean this audit had sat in the queue for over a week, and remained in the queue even after the question was actually closed. If that is what's going on, hopefully this meta post (and the audit it documents) will draw renewed developer attention to the problem of stale review audits: specifically, that they incentivize robo-reviewing and punish us when we pay attention, the exact opposite of what audits should do.
    – Eliah Kagan
    Commented Apr 9, 2015 at 3:24
  • I agree some of the audit questions are odd. Perhaps the audit questions need to be audited first - Is this a good question for auditing ? Did the community take the correct action ? Sometimes "the community" is too fast to close good questions, typically as a dupicate
    – Panther
    Commented Apr 9, 2015 at 13:31
  • @bodhi.zazen I agree that might be useful. It wouldn't have helped in this case though: I failed this audit for performing the same action that the community had performed, i.e., for voting to close a question as a duplicate of the question it was already duped to.
    – Eliah Kagan
    Commented Apr 9, 2015 at 14:19

0

Reset to default

You must log in to answer this question.