2021: a year in moderation - Handled Question Flags

Question

A question (but first a comment) regarding 2021: a year in moderation:

First, that's 32,646 actions taken by our hard working Mod team in 2021. I know that there's even more that you do that isn't even represented in these stats -- Thank you! You are definitely appreciated!

On the "Community" side, I do want to question one stat.

This is repeating a point I made when I sanity-checked the Stack Overflow results, but since it's related to a chat in AUGR that a few of us (@Zanna, @RandomPerson, and others) had back in September, I figured I'd check the AU numbers as well.

From AUGR chat:

Zanna: I mean, I agree with @RandomPerson that reviews automatically getting invalidated due to nobody ever doing them is real real bad

At that time, the concern was primarily over the First Posts Review queue (now, of course, split into First Questions and First Answers). But the "Close Vote Review" queue is, in my understanding, fed partly (mostly?) from Flagged questions.

In these AU moderation stats, we see that:

• A total of 14,011 questions were flagged either by Mods or by Community Members (or Community Bot).
• A total of 13,646 question flags were "handled" (97.39%)
• Leaving, presumably, only 365 question flags (2.61%) that weren't handled in 2021.

But are things really that good? According to a comment from @Makyen (mod) on my meta post on SO:

Close-flags, like close-votes, do age out but those flags are not presented to moderators. They go to the CV review queue.

So are we (the community) really "handling" nearly every question flag?

Zanna pointed out a SEDE query that checked whether a review had "aged out" due to expired votes, and I've attempted to adapt that to Close Votes Reviews. I've extended the range of days it was in the queue to:

• Between 4 and 7
• Between 14 and 17

... based on this information that Close votes expire at a rate of 1 per day after day 4 (for highly viewed posts) and day 14 (for low-eyeball posts).

If the query assumptions are accurate, we actually had 4,037 questions here on AU that (likely) aged out of the Review queue in 2021. That's almost 30% of the ones that the stats above say that we (the community) "handled."

IMHO, that's hiding a rather significant problem that, as Zanna said, "is real real bad".

Related side note: It seems very difficult to get a Close Vote Review to muster enough votes (either "Close" or "Leave Open", really), even in the most obvious of cases. Even when the OP says in the comments that the problem was a typo. Even when the OP agrees in the comments that the problem isn't Ubuntu related and reposts on Super User.

So:

• Are "invalidated"/"aged out" reviews being counted as "Handled" in the Moderation Stats just because they moved from "Flagged" to "Review Queue"?
• Should they be? Or should the stats instead show that there's a potential issue with Review Queues not being "handled"?

Answer 1

A flag to close a post is considered to be handled if it is marked helpful. This happens if the post receives at least one close vote, regardless of the outcome of the close review.

For example, you flagged Ubuntu 20.04 disappears from Windows 10 WSL2 after reboot for closure, and another user subsequently voted to close it (I can see this in the post timeline, but it is not visible to non-mods, frustratingly). The close vote aged away (incidentally, the close vote was not cast from the queue). One leave open vote was cast, but that would not remove the post from the queue, so a strict perspective would consider the close effort unhandled. However, the flag was marked helpful, and so it was "handled".

A flag to close is also considered handled if it is declined, which I think only happens if and only if the post receives no close votes AND the final outcome of the review is leave open (this requires 3 leave open votes*). This is okay - we can consider the close effort handled there, I think.

To be clear:

• A single close vote cast on a post flagged for closure instantly handles the flag as helpful.
• If the close review is resolved as leave open before any close votes are cast, any flag to close on it is declined.

Given that the post receiving only one close vote is sufficient to handle close flags, I am not surprised that most such flags do get "handled", although the post does not really get handled in many such cases.

Just a thought - I would be surprised if posts primarily came to the Close Votes queue from flags - I think they are at least as likely, if not more likely, to arrive there because they receive a close vote from a user with at least 3k reputation. When one reviews the CV queue, one can see the previous close votes the question received. If no votes are seen, the post only has a flag. I feel this is rare - the case for less than 1 in 20 posts, while having only 1 previous close vote is common.

_{* that is, 3 leave open votes by normal users - a mod voting to leave open at any point would also complete the review, automatically declining any unhandled flags to close}