13

Recently Luis went SPACESHIP and brought the 1000-or-so items in the review close queue down to zero. A truly epic effort that I'm sincerely grateful for.

Shortly after, he posted on Meta about his frustrations with the number of questions in there that had already been answered (in some cases years ago) that people were still wringing their hands over, or voting to close (even brand new questions) without having asked for clarification or detail.

What he got in return was a visit from the Meta Police. A local cop pulled in support from MSO/MSE and they ended up in a mix between a syntactical argument, an "Oh but the rules —insert link to 3yo meta posts— say this" and a fight over what the system is supposed to do. The whole exchange dissolved into nonsense and was deleted.

Deliberately or otherwise, everybody replying missed the very important point Luis was making.

I wanted to reprise the post in some form because I have been saying similar things on this topic for years. Experienced users shouldn't find it surprising: The review system puts so much emphasis on coddling old, already-dealt-with questions that it's suffocating us. I'm not saying it can't ever work but it relies on people and we don't have a limitless supply of those.

I think some people struggle to understand the scale of the issue. At the moment we are failing to answer or close 3000-5000 questions a month. That's our priority here.

Instead of doing that though, we have people submitting poor close votes, each of which uses up to another 4 votes from other users. We average around 40 seconds a review so including the initial submissions, we can pour over three minutes into something. Longer if people are careful. Even longer if that goes to meta. Even to leave stuff open (which in my experience is at least a third), we're committing vast amounts of people-time.

This adds up. 40 seconds per review, ~150 reviews a day. That's 50 hours a month going on review.

I'm certainly not saying all review is bad (and neither did Luis) but if you only take one thing away from this exchange, let it be that you should just be more thoughtful about what it is you're submitting into the review system. If it's old and answered, really ask yourself if that deserves taking another 4 people off today's questions.

It has been mentioned that the reason a lot of old questions float back into review circulation is that somebody or something bumped them onto the front page. The solution for this is really very simple, don't use the front page. Hang out on the New Questions page and you'll be giving most of your attention to the right questions.

5
  • 5
    I had always assumed that the close queue was sorted by date, so that more recent questions get priority. Is that actually the case? If not, perhaps it should be - it would be a very simple way to achieve prioritisation of new questions over old.
    – bain
    Aug 7, 2014 at 16:58
  • @bain meta.stackexchange.com/a/199912/213575
    – Braiam
    Aug 7, 2014 at 17:02
  • 1
    So that's saying they're just sorted by how they're reviewed. First by number of reviews, descending and then by time added to review, ascending. The age of the question doesn't appear to factor into things but I agree that wouldn't be a bad first step.
    – Oli Mod
    Aug 7, 2014 at 17:37
  • 2
    If answering questions is more important than closing them, and there is a constant stream of unanswered questions (~4000 a month), then the implication is that nobody should be reviewing the close queue when they could instead be answering questions. Which means that nobody should ever review the close queue.
    – bain
    Aug 7, 2014 at 17:55
  • New duplicates are the exception.
    – Oli Mod
    Aug 7, 2014 at 17:57

3 Answers 3

3

The review system puts so much emphasis on coddling old, already-dealt-with questions that it's suffocating us

Your premise is entirely wrong, so at first I didn't touch it, thinking, that it isn't worth my time, but here goes:

It doesn't, at all. In fact, the review system gives priority to newer posts at certain points of the day, rather than the older ones and always gives priority to the ones that has more reviews, is a pseudo-random way. The fact that you see old crap in the review system is because the lack of reviewers which I addressed in another post how to improve it.

I think some people struggle to understand the scale of the issue. At the moment we are failing to answer or close 3000-5000 questions a month. That's our priority here.

Remember that the system is optimized for pearls. But right now, in this very same moment, we are not even getting sand to produce pearls, which is really a problem. Anyone naive enough to not recognize it, is hurting the site.

So, how do you think we fix the problem? Well, users find that removing old cruft, bad quality content, which is vague, inaccurate, is a good way to solve the problem at hand, why are moderators against it? I find it completely counter productive "skimming through a lot of duplicates (questions) finding helpful information nuggets" that vague questions provide. Again, optimize for pearls, not for sand!

Everyone knows I'm the first to ask for a question that was incorrectly closed to be reopened and even answer them, but all the cases presented so far are sand, paired with more sand. Low quality content that should not exist in the first place, and will hopefully be deleted in the posterity, so I expect less posts of this kind in the future.

Now, generally speaking, someone defending this kind of content (old, low quality crap) is doing a disservice for Ask Ubuntu. Old questions or not, looks like crap, we don't want it on the site. Doing a house cleaning starting from the top to the bottom is a sane behavior, nobody needs to stop it.

Ah, BTW, I don't recommend anyone to watch only the "new questions" feed, I don't do that at all. I get depressed and stop answering questions. Instead I find old questions that needs love, it's even more rewarding. Maybe we should get implemented the "Recommended" tab, but that is another bundle of problems due the general state of our tags.

7
  • 4
    You're advocating spending more time on removing questions instead of answering things. That's my problem and that's what you're not understanding. Haven't you ever done any project management before? Our users and their time is finite. I'd love to have enough people to double-review every question and remove all the crap but that isn't the reality we're working in.
    – Oli Mod
    Aug 7, 2014 at 17:42
  • @Oli "You're advocating spending more time on removing questions instead of answering things" I don't know what to say... but I'm suspecting you don't read my post at all. Read my last paragraph and tell me where you got it wrong.
    – Braiam
    Aug 7, 2014 at 18:07
  • 3
    Given your entire post is a misrepresentation of our arguments, you'll have to forgive my approach but your first four paragraphs are about deleting old crap. Something can only quack so often before I call it a duck. Nobody is defending old crap however I am advocating on focussing all our efforts on the new posts (crap and otherwise) and to let nature run its course on the old stuff.
    – Oli Mod
    Aug 7, 2014 at 18:39
  • @Oli "let nature run its course on the old stuff" well, I would love to, but it doesn't
    – Braiam
    Aug 8, 2014 at 18:49
  • 1
    Come on. That's a recent bug in the system. There's no suggestion it isn't being fixed. (edit: it's running again now)
    – Oli Mod
    Aug 8, 2014 at 19:28
  • 5
    @Braiam Citing a breakage in the system like that is just silly. Tons of old questions are automatically deleted all the time. If we catch up with the new junk the system will take care old and we'll have a clean site. Imagine that!
    – Seth
    Aug 10, 2014 at 0:39
  • @Seth I'm only putting in contrast that the system doesn't always work how we expect them, this is more evident when Shog proposes more aggressive deletion criteria.
    – Braiam
    Aug 10, 2014 at 23:07
0

The problem is that the question handling process does not have an efficient pipeline for moving large numbers of questions to a closed state. There are tens of thousands of useless old questions still hanging around, waiting for a user to find them and add a comment or answer or edit, which brings the question back to the front page, which promotes further activity. Users are encouraged to flag all questions, regardless of age. Asking users to change this behaviour is only going to have a limited effect - the behaviour is just a symptom of the dynamics of the underlying system. To get meaningful change, you would have to change the system.

The lifecycle of a question is:

  • User has a problem and finds AskUbuntu
  • User creates a new question
    • If question stays on the front page for long enough it attracts votes and comments
    • If it gets bumped from the front page, question is less likely to attract any answers
  • At this point, either:
    • (rare) User accepts answer
      • Question closed
    • (usually) User disappears, leaving the question open
      • Loop forever until flagged for closure:
        • Time passes
        • Someone finds the question and adds a comment/edit/answer
        • Question hits the front page and others add comments/edits/answers
      • Question enters close queue
      • Question closed.

The implication of this is that the lifecycle for most questions is to end up in the close queue.

The solution employed by others facing similar problems is to streamline the process, and impose strict criteria that enable automatically classifying and closing issues. e.g. Launchpad's automated bug closing mechanism ("this bug was reported a while ago and there hasn't been any activity in it recently. We were wondering if this is still an issue?") and the automated duplicate detector that matches regular expressions in log files. Similar strategies could perhaps be used to manage the question handling process of AskUbuntu.

6
  • 3
    Your diagram isn't quite accurate. Any answer with a positive score kicks the question off the "unanswered" pile. There is no requirement that every question ends up either closed or accepted... Just a good answer will do.
    – Oli Mod
    Aug 7, 2014 at 11:35
  • In the larger argument about improving the system... I agree (and there are a few changes I'll be recommending to SE) but there are relatively few users generating the flags that lead to all this review churn; this is something that education can help with.
    – Oli Mod
    Aug 7, 2014 at 11:39
  • 1
    Yes, a question can get marked "answered" by having a single answer with a single upvote, but this does not end its life - people will still comment on it, try answering it, etc. Eventually the question will probably become irrelevant, the release the question was asked about will become EOL, and then someone might flag to close it. This is more likely if it attracts poor quality answers (ie. an upvote for a friendly answer that is incorrect or not good enough). The system allows this to happen. From reading meta it seems this is an issue that comes up again and again.
    – bain
    Aug 7, 2014 at 13:53
  • 2
    You're still worrying about yesterday's jam. Old unanswered posts existing isn't the problem, it's that we're all spending an inordinate amount of time dealing with them. This goes doubly for things being forced in front of reviewers. I'd like to stop that happening so that answerers and reviewers alike can focus on the hundreds of new posts we get every day. If your ADHD won't let you allow ancient questions to float past on the front page, live in the stream. Trust me, it's a much happier existence.
    – Oli Mod
    Aug 7, 2014 at 15:50
  • 1
    If old unanswered posts did not exist, or could not be closed, then no time would be spent dealing with them. All I'm saying is that changing the system would be a lot more effective than asking people not to do the things that they have previously been encouraged to do. This is not a new problem. The previous consensus was that abandoned questions should be flagged for closure - you implied so. Perhaps that does not scale.
    – bain
    Aug 7, 2014 at 17:38
  • Indeed, scale bites us all eventually. That "you implied so" post was actually a problem of scale itself. There weren't enough moderators to handle the rapidly increasing number of flags. The very same is now true in the review queue.
    – Oli Mod
    Aug 7, 2014 at 17:44
-3

The review system puts so much emphasis on coddling old, already-dealt-with questions that it's suffocating us.

Indeed, the review system is a contradictio in terminis:

  1. Closing a question gets it 'out of the site' so that we do not spend time and energy answering it. Also so that people don't get angry for not getting an answer.

  2. In order to close it, 5 expert users (in theory) should pay attention to the question and vote democratically.

In short, closing a question actually gives it a lot more attention than one wants to give it.

Ignoring old questions that should be closed is not going to help it and goes against the gut feeling of 'wanting to do things correctly' and 'not leaving a mess behind'.

A proposal (feel free to disagree, of course). The system should be changed structurally.

  1. The close review queue should be removed altogether. It is not working anyway. Questions can only be closed when actually visiting the question page.
  2. The close votes can be taken away when a user abuses them or fails to welcome the user and ask for more information. A tool is made for moderators to check close vote activity.
  3. If a CV review queue is upheld, then reviewers should get the questions to review that are in the tags they follow! (Here, SE take that, for free) It is easy to be demotivated when the close review queue just keeps on growing (been there, had that feeling). Most frustrating is that you have to read a whole question, then realize you do not know enough about the topic and thus you have to skip. If I get to review python, django, wget, etc. questions only that will help a lot.
5
  • 3
    You guys are entirely missing the point. We aren't saying "Don't vote to close old junk", just be careful when you do do it. Too many people were closing old 11.10 questions as EOL when they had been long since answered. That is pointless and harmful.
    – Seth
    Aug 10, 2014 at 18:19
  • I beg your pardon? "You guys"? "missing the point?" In what way is my point contradicting yours?
    – don.joey
    Aug 10, 2014 at 18:29
  • @Seth Ok. Moderator downvote. No reaction afterwards. Way to go. Next time I'll just not bother anymore. It is also very typical that we get a full theoretical expose to hear afterwards what is really the problem: people CVing questions that are now EOL. That is not part of the OP.
    – don.joey
    Aug 12, 2014 at 6:04
  • You were going to get a reply, I just haven't hide time to write it out yet. Patience.
    – Seth
    Aug 12, 2014 at 15:06
  • By "you guys" I meant every user that has tried to interact with the past two questions that have been posted about this have entirely missed the mark. You all seem to think we are saying "ignore old questions that need to be closed" no one is saying that. Downvote because I think your proposal is bad.
    – Seth
    Aug 15, 2014 at 17:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .