How to know which questions should be closed, a reprise

Question

Recently Luis went SPACESHIP and brought the 1000-or-so items in the review close queue down to zero. A truly epic effort that I'm sincerely grateful for.

Shortly after, he posted on Meta about his frustrations with the number of questions in there that had already been answered (in some cases years ago) that people were still wringing their hands over, or voting to close (even brand new questions) without having asked for clarification or detail.

What he got in return was a visit from the Meta Police. A local cop pulled in support from MSO/MSE and they ended up in a mix between a syntactical argument, an "Oh but the rules —insert link to 3yo meta posts— say this" and a fight over what the system is supposed to do. The whole exchange dissolved into nonsense and was deleted.

Deliberately or otherwise, everybody replying missed the very important point Luis was making.

I wanted to reprise the post in some form because I have been saying similar things on this topic for years. Experienced users shouldn't find it surprising: The review system puts so much emphasis on coddling old, already-dealt-with questions that it's suffocating us. I'm not saying it can't ever work but it relies on people and we don't have a limitless supply of those.

I think some people struggle to understand the scale of the issue. At the moment we are failing to answer or close 3000-5000 questions a month. That's our priority here.

Instead of doing that though, we have people submitting poor close votes, each of which uses up to another 4 votes from other users. We average around 40 seconds a review so including the initial submissions, we can pour over three minutes into something. Longer if people are careful. Even longer if that goes to meta. Even to leave stuff open (which in my experience is at least a third), we're committing vast amounts of people-time.

This adds up. 40 seconds per review, ~150 reviews a day. That's 50 hours a month going on review.

I'm certainly not saying all review is bad (and neither did Luis) but if you only take one thing away from this exchange, let it be that you should just be more thoughtful about what it is you're submitting into the review system. If it's old and answered, really ask yourself if that deserves taking another 4 people off today's questions.

It has been mentioned that the reason a lot of old questions float back into review circulation is that somebody or something bumped them onto the front page. The solution for this is really very simple, don't use the front page. Hang out on the New Questions page and you'll be giving most of your attention to the right questions.

Answer 1

The review system puts so much emphasis on coddling old, already-dealt-with questions that it's suffocating us

Your premise is entirely wrong, so at first I didn't touch it, thinking, that it isn't worth my time, but here goes:

It doesn't, at all. In fact, the review system gives priority to newer posts at certain points of the day, rather than the older ones and always gives priority to the ones that has more reviews, is a pseudo-random way. The fact that you see old crap in the review system is because the lack of reviewers which I addressed in another post how to improve it.

I think some people struggle to understand the scale of the issue. At the moment we are failing to answer or close 3000-5000 questions a month. That's our priority here.

Remember that the system is optimized for pearls. But right now, in this very same moment, we are not even getting sand to produce pearls, which is really a problem. Anyone naive enough to not recognize it, is hurting the site.

So, how do you think we fix the problem? Well, users find that removing old cruft, bad quality content, which is vague, inaccurate, is a good way to solve the problem at hand, why are moderators against it? I find it completely counter productive "skimming through a lot of duplicates (questions) finding helpful information nuggets" that vague questions provide. Again, optimize for pearls, not for sand!

Everyone knows I'm the first to ask for a question that was incorrectly closed to be reopened and even answer them, but all the cases presented so far are sand, paired with more sand. Low quality content that should not exist in the first place, and will hopefully be deleted in the posterity, so I expect less posts of this kind in the future.

Now, generally speaking, someone defending this kind of content (old, low quality crap) is doing a disservice for Ask Ubuntu. Old questions or not, looks like crap, we don't want it on the site. Doing a house cleaning starting from the top to the bottom is a sane behavior, nobody needs to stop it.

Ah, BTW, I don't recommend anyone to watch only the "new questions" feed, I don't do that at all. I get depressed and stop answering questions. Instead I find old questions that needs love, it's even more rewarding. Maybe we should get implemented the "Recommended" tab, but that is another bundle of problems due the general state of our tags.

Answer 2

The problem is that the question handling process does not have an efficient pipeline for moving large numbers of questions to a closed state. There are tens of thousands of useless old questions still hanging around, waiting for a user to find them and add a comment or answer or edit, which brings the question back to the front page, which promotes further activity. Users are encouraged to flag all questions, regardless of age. Asking users to change this behaviour is only going to have a limited effect - the behaviour is just a symptom of the dynamics of the underlying system. To get meaningful change, you would have to change the system.

The lifecycle of a question is:

• User has a problem and finds AskUbuntu
• User creates a new question
  • If question stays on the front page for long enough it attracts votes and comments
  • If it gets bumped from the front page, question is less likely to attract any answers
• At this point, either:
  • (rare) User accepts answer
    • Question closed
  • (usually) User disappears, leaving the question open
    • Loop forever until flagged for closure:
      • Time passes
      • Someone finds the question and adds a comment/edit/answer
      • Question hits the front page and others add comments/edits/answers
    • Question enters close queue
    • Question closed.

The implication of this is that the lifecycle for most questions is to end up in the close queue.

The solution employed by others facing similar problems is to streamline the process, and impose strict criteria that enable automatically classifying and closing issues. e.g. Launchpad's automated bug closing mechanism ("this bug was reported a while ago and there hasn't been any activity in it recently. We were wondering if this is still an issue?") and the automated duplicate detector that matches regular expressions in log files. Similar strategies could perhaps be used to manage the question handling process of AskUbuntu.

Answer 3

The review system puts so much emphasis on coddling old, already-dealt-with questions that it's suffocating us.

Indeed, the review system is a contradictio in terminis:

1. Closing a question gets it 'out of the site' so that we do not spend time and energy answering it. Also so that people don't get angry for not getting an answer.

2. In order to close it, 5 expert users (in theory) should pay attention to the question and vote democratically.

In short, closing a question actually gives it a lot more attention than one wants to give it.

Ignoring old questions that should be closed is not going to help it and goes against the gut feeling of 'wanting to do things correctly' and 'not leaving a mess behind'.

A proposal (feel free to disagree, of course). The system should be changed structurally.

1. The close review queue should be removed altogether. It is not working anyway. Questions can only be closed when actually visiting the question page.
2. The close votes can be taken away when a user abuses them or fails to welcome the user and ask for more information. A tool is made for moderators to check close vote activity.
3. If a CV review queue is upheld, then reviewers should get the questions to review that are in the tags they follow! (Here, SE take that, for free) It is easy to be demotivated when the close review queue just keeps on growing (been there, had that feeling). Most frustrating is that you have to read a whole question, then realize you do not know enough about the topic and thus you have to skip. If I get to review python, django, wget, etc. questions only that will help a lot.