0

Dear moderators this message is for you. Some weeks ago there was a hike in the number of audits conducted in the review queue. Unfortunately it has created more problems than it solved, more specifically:

  1. Audits have become a large share of the review tasks, in my case about one third of the review tasks are tests.

  2. Many tests seem to be unclear or at least prone to ambiguous actions, creating a good deal of backlash.

  3. A bug exists that allows users to identify tests - clicking on edit in a test brings up the message "This post is deleted [sic]". I am now able to avoid all audits using the edit button.

In light of these problems I would suggest the following procedure:

  1. Suspend review audits immediately.

  2. Fix the bug with the edit button to prevent users from identifying audits.

  3. Remove all ambiguous tests.

  4. Re-introduce bug-fixed audits but on a lighter volume, say, one test per twenty reviews.

Stop. Look and Listen. The audit system is not working properly.

11
  • I think the audit system is the same on all sites, so this Q on meta.su is relevant. I also thought I was getting too many but it turns out they're random.
    – terdon
    Commented Mar 9, 2014 at 15:45
  • 1
    Anyone that studied Statistics knows they are not random. Just keep a log of the tests that show up, it will tend to a specific proportion of the review tasks on a daily or weekly basis.
    – Luís de Sousa
    Commented Mar 9, 2014 at 17:29
  • 1
    You cannot know if they're random with the data you have. To test, you will need the network-wide statistics, how many reviews and how many edits and then run a phyper or x2 or similar test. Even if you get 10 audits in 10 reviews, you cannot conclude that they are not random.
    – terdon
    Commented Mar 9, 2014 at 17:34
  • 1
    Hi terdon, polling is well studied method by which you study a whole population through a small sample.
    – Luís de Sousa
    Commented Mar 10, 2014 at 7:39
  • Yup, but that sample's size needs to be more than 1 if you want to reach any significant conclusion. With a sample of one, you might calculate a probability of randomness but cannot conclude one way or another.
    – terdon
    Commented Mar 10, 2014 at 15:18
  • also: meta.stackexchange.com/questions/179651/…
    – Mateo
    Commented Mar 10, 2014 at 17:43
  • @Mateo that's more relevant for close votes. OP only has access to First Post and Late answers.
    – Braiam
    Commented Mar 10, 2014 at 22:22
  • @Braiam well then...
    – Mateo
    Commented Mar 10, 2014 at 22:24
  • While there is no doubt this system is needed, it's also flawed. In reviewing a first answer I have started opening the question in another tab so that I can understand the answer in context and determine whether it is indeed a good question. Today, I followed that process and was flagged with the above FAIL for attempting to leave a comment that pointed out that it was an exact duplicate of an already accepted answer. This would be amusing if it wasn't so sad. Fighting "Robo reviewing" with "robo audits" seems contra-indicated.
    – Elder Geek
    Commented Jun 20, 2014 at 14:30
  • Of course the lesson learned from this is to accept blatant copying of accepted answers as perfectly acceptable. (symphony of sarcasm detected).
    – Elder Geek
    Commented Jun 20, 2014 at 14:33
  • The steady increase of these tests made me stop reviewing because I felt I was wasting my time when "passing" so many tests instead of actually helping.
    – Delorean
    Commented Aug 24, 2017 at 3:42

1 Answer 1

Reset to default
4

The audits will not go away. They are in place due several complains about reviewers not reviewing or misusing the review queues. Summary, reviewers bring this forth by themself. Now, if you seriously fails so many review audits in good faith, then go outside the queue and fix the issue you found!

Downvote, vote to close, edit, flag, etc. That's why there are many "ambiguous tests" due our inconsistency when evaluating the quality of a post. Things that should be closed are upvoted, things that should be upvoted are closed (this is just figurative, I haven't found a post in that state), spam is not getting flagged, etc. So, if you find that you failed an audit and don't agree with it, present an specific case so people can fix it.

Now, addressing your specific concerns:

Audits have become a large share of the review tasks, in my case about one third of the review tasks are tests.

Audits are totally random, is just bad luck that you get so many of them. I've had only ~20 out of all reviews I've done so far.

Many tests seem to be unclear or at least prone to ambiguous actions, creating a good deal of backlash.

I believe you should then dispute the review audits in the same fashion everyone else does. Post the specific audit you failed and why you disagree with it. Otherwise is just rambling.

A bug exists that allows users to identify tests - clicking on edit in a test brings up the message "This post is deleted [sic]". I am now able to avoid all audits using the edit button.

I'm sure that this is a known issue (can't find it in MSO), but most of deleted post are unsalvageable for anyone but OP so maybe you should instead cast flags over them.

Stack Exchange team is observing how audits behave (at least that's what they made us think) and will be able to fix any issue it's present. Again, if you feel that your action was correct due the state of the post, go and fix it, don't just continue without fixing the problem.

6
  • 2
    Exactly. We're the bug, not the audit system.
    – Seth Mod
    Commented Mar 9, 2014 at 15:33
  • Seth you are not listening. An audit system that is this easy to circumvent serves no purpose.
    – Luís de Sousa
    Commented Mar 9, 2014 at 17:32
  • 1
    @LuísdeSousa You are missing the point of the audit system. It's only in place to assure us that reviewers are actually paying attention to their reviews. That is it's only purpose. It should be easy!
    – Seth Mod
    Commented Mar 9, 2014 at 18:44
  • @LuísdeSousa quoting Shog words: Remember: review audits are meant to be helpful, a tool for teaching new reviewers what is expected and maybe staving off fatigue in experienced reviewers.
    – Braiam
    Commented Mar 9, 2014 at 19:02
  • 1
    @Seth that’s exactly what several of them are failing at - you get a "Stop..." when you don't do exactly what was envisioned by the designers - but that doesn't mean you didn't pay attention.
    – guntbert
    Commented Mar 9, 2014 at 20:56
  • 5
    @guntbert I'm sorry but I really don't under why everyone makes such a big deal out of it. Audits aren't perfect. Period. Even moderators fail audits sometimes, they aren't perfect. Better question moderation will lessen the fails on odd questions, but overall everyone is going to fail on audits occasionally and it takes the audit system a bit to get used to us. Stop worrying about it, if you fail an audit roll your eyes and move on.
    – Seth Mod
    Commented Mar 9, 2014 at 21:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .