7

Over on Meta Stack Exchange, someone's reported having received the same post as a review audit in the First questions review queue five times overall. Per this post from a then-employee, when the system generates a review audit, it does not check to see if a given user has previously received the same post as an audit, and just randomly selects an eligible post to generate an audit from.

That the same post was randomly selected as a known-bad audit for the same user five times leads me to believe that the pool of eligible known-bad audit questions is extremely small: per the probability rules, the chances of the same post being selected are one in the number of eligible questions to the fifth power. (Unfortunately, as the post is deleted, I as a <10k user cannot see its timeline, and so don't know how many times it was used as an audit in total, for all users.)

Why is the selectable pool of known-bad first question audits so small, and what can be done to increase it? Are there some classes of questions being excluded from selection (e.g. those asked more than one year ago)?

I'm asking this because even if the feature change requested in the linked report, to check to see if the same post was used to audit the same user, is implemented, this could lead to a dearth of available questions to select once a user has audited all the eligible questions, which means they're no longer being checked for robo-reviewing.

4
  • 1
    It was used 26 times in total so far. It first appeared as an audit task in jan 15.
    – Dan
    Feb 5, 2022 at 14:26
  • 3
    @Dan Can you please link to a screenshot for <10k users, as they just see a 404 when clicking on that link?
    – gparyani
    Feb 5, 2022 at 17:58
  • 1
    I would guess that this has something to do with the redesign of the queues and perhaps these queues being so new. I guess only developers would know though?
    – Zanna Mod
    Feb 5, 2022 at 18:17
  • 2
    @gparyani I was on my phone earlier so I couldn't take that screenshot. Can't fit the whole thing in one xD But here it is: i.stack.imgur.com/yl3Bm.png
    – Dan
    Feb 7, 2022 at 7:44

1 Answer 1

8

We modified the review queues so that users cannot be audited on questions they already reviewed. We hope that solves your issues!

Answering your question about the question pool - yes, we audit users only on recent questions.

1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .