28

In the fashion of the discussion on meta.SO about unanswered questions on Stack Overflow

Currently, out of 120,000 odd questions, only about 70% of them are considered answered. But it wasn't this bad some time ago. Last year, after our cleanup event, our answered rate went as high as 86%. And that stat has been on the downfall ever since.

Here's some number crunching to get the idea:

Percentage of open questions with answers, by quarter

Percentage of open questions with answers, by quarter

The number of questions we get have significantly increased since 2012 but the answers haven't caught up with them yet. Actually, they are not even trying to catch up.

Question quality by quarter

Question quality by quarter

Likewise, the number of questions that have received at least one vote have been on the decline as well. But there haven't been much increase on downvoted questions. I think votes should be used to quickly sort good content to the top and bad content to the bottom. If we are to club decent quality questions with poor quality content by not voting on either of them, we are doing no justice to the system. Anyway, that probably should be raised in a separate discussion.

Question Votes by Quarter

Question Votes by Quarter

People predominantly prefer to upvote questions or not vote at all. Just 6% of the votes are downvotes. Maybe we shouldn't be so averse to the downvote button.

Now that I have established all of that, what we can do about the unanswered questions? Should we be concerned yet?


Me being a little concerned about the state of the site, I have a few ideas that I would like to throw here for discussion. You are strongly encouraged to brainstorm a few more ideas and discuss the validity of the ideas raised here. The answer section is open for all.

  1. Enforcing a stricter policy on content quality

    I believe we should consider actively putting questions that don't have adequate details and information that are needed to provide a satisfactory answer "on hold". Historically, we were quite lenient on questions that didn't have enough details and didn't close them right away on the hopes that OP would edit the details in.

    But this workflow is horribly broken and more often than not, that's exactly what is not happening. Questions lie around on the site for months together without enough information to answer them. And OP will not be active either. Maybe we should start to put them "on hold" first and then reopen them when they have enough details. This workflow should work better since the system now automatically adds recently closed and edited questions in the reopen queue.

  2. Restricting the asking privileges to registered users

    I think this little tweak to our system would do a world of good. For a start,

    • it would keep away a decent chunk of spam posts
    • prevent drive-by users from posting their questions and then fleeing the site
    • tremendously decrease the chance of a user losing access to their cookie-based accounts thereby avoiding anonymous edits from OP or non-answers posted by OP from another account to provide further information.
  3. Downvoting incomplete questions so they fall to the bottom

    This probably would help on multiple folds.

    Now, if you are one of those good people not comfortable with downvoting questions because "it is not the right/best/kind/nice thing to do", I think we should start putting the best interests of the community first instead of sacrificing the community at all.

  4. Inviting and encouraging 'experts' to answer questions

    It is a known secret that most of the questions we receive right now are either very mundane or highly specific. Maybe all we need is more people in each of the specific field answering those questions.

    Let me end of with another data query showing how askers compare with answers..

    Number of users answering or questioning

    Number of users answering or questioning


Answering Oli's concerns.

jokerdino's first three suggestions have problems:

  • Our existing policies stipulate quality... People ignore policy.
  • Forced registration won't float with SE and doesn't solve the problem of people asking bad questions.
  • Downvoting is just another way of cleaning. It's still time taken from members to review something.

We should always be trying to attract people in.

  • People may ignore policy. But that's not an argument for never enforcing one. Skeptics.SE expects a remarkably high standard (in comparison with the rest of Stack Exchange network sites) from their users and the general quality of the posts over there would tell you the fruits of enforcing such a quality policy.

  • Forced registration is not a new concept, not to Stack Exchange. Stack Overflow has it for a long time and Programmers.SE has required registration for quite some time now. And no single solution will overnight solve the problem of people asking bad questions and requiring registration will help prevent people easily losing access to their accounts and questions. This will be beneficial especially with mods no longer being able to merge user accounts at our discretion.

  • Now I am not sure why we are so averse to any form of cleaning. Downvoting is a form of cleaning (actually, I would call it sorting) and that's pretty much the reason why it's even there. And downvoting will help highlight good content. It may take some time from the reviewers (realistically speaking, voting costs you may be 10 seconds but the benefits are far reaching) but it will definitely help signal to the general audience that a post is not great.

When we are getting 180+ questions on average everyday, I am sure we can afford to keep some of the questions away. It is as much about keeping our community happy and healthy as it is about getting more people in.

  • I know I can do it, but there is a table showing the total questions/upvotes-downvotes relationship? Sometimes we don't even vote the questions up-down and just comment. – Braiam Aug 16 '13 at 18:27
  • 2
    Is there a way we can ask SE to crank up our close votes? I can burn through all of mine in about 10 minutes a day and I've been maxing that out for a while. – Jorge Castro Aug 17 '13 at 14:40
  • @Braiam You mean like how many questions get voted upon? – jokerdino Aug 18 '13 at 6:58
  • @jokerdino yeah, but let me sync myself: of all the questions (100%) how many have no votes, how many have upvotes, and how many have downvotes. – Braiam Aug 19 '13 at 0:19
  • I don't think the issue is the number (or percentage) of unanswered questions, really. As long as we can effectively use the search and locate a meaningful answer I don't worry how much cruft exists. Assuming SE's infrastructure can continue to store and index it the issue is a moot point. Put differently, relevant search results matter far more than % unanswered. – Tom Brossman Aug 21 '13 at 21:16
  • @TomBrossman what we would like is not wasting time into looking for the right answer (it takes me 3-10 minutes deciding if the answer I found apply to the particular case, just to be being told that the user already tried those) when users could be more proactive and look for themself (SEO of SE sites are pretty awesome at that) while we deal with questions that really has no answers (which are about 4/30, personal stats). – Braiam Aug 22 '13 at 1:23
14

When people mention cleanup and our "answer rate", they're talking about a historical number. A number that I frankly feel isn't that important. It's a stat that we can manipulate if we close a big bunch of questions.

The important number is the rate we're dealing with new questions. We need to deal with the questions that are coming in faster, or lower the number. jokerdino's first three suggestions have problems:

  • Our existing policies stipulate quality... People ignore policy.
  • Forced registration won't float with SE and doesn't solve the problem of people asking bad questions.
  • Downvoting is just another way of cleaning. It's still time taken from members to review something.

We should always be trying to attract people in.

I have a theory...

We're spending 13 hours every day fixing questions missing vital information

On any given day, ~50% of the questions will be about hardware in some capacity. 80% of these need additional detail that we have to tease out of the user and a good 10-20% of those don't respond. Either way, we're sinking 5-10 minutes into each of these questions just to make them answerable. Collectively that's 13 hours.

4-8% of our new questions are unanswerable, abandoned crap. That's over an hour every day completely wasted.

And because these questions are so stop-start, they completely bypass the useful stages of the review process. By the time we can work out that it's a dupe, everybody has forgotten all about it.

What do you think the opportunity cost of 13 hours is? Let me put it another way: How many questions do you think you could answer in 13 hours? If we could ensure that every question were answerable, I reckon I could get to all of them in 13 hours. And that's just the time we regain.

Cleanup is not our saviour

So going back to cleanup. The few sprints we've done have been great at improving our historical "answer rate" but they do nothing to improve our current answer rate. If anything, they're diverting resources (people) from answering new things.

Additionally there are some questionable behaviours and side effects cleanup brings like closing good questions because they're too hard for us at the moment and voting for crappy answers so the question is technically "dealt with". When we do this... Well I think I said it best in the moderator chat room:

Closing something because we (as a site) can't answer it just compounds one failure on top of another.

Cleanup is a very blunt tool that lets us hide how bad we've been at keeping on top of things in realtime. It's a last resort.

So here's one way to ensure that new hardware questions have all the detail we require... A big-bearded, lightning-bolts-and-all WIZARD!

When somebody goes to ask a new question, before they see a text box we ask them a few questions to determine where their problem is.

  • Is you question about the following problems?
    • Blank screen on boot
    • No wireless connection or other networking issues
    • The resolution on my screen isn't correct
    • I don't know how to install the right graphics driver
    • I can only see the desktop
    • Nope, just take me to the form

This list needs to represent four or five of the most common and hardest to diagnose issues. The above are just examples.

Let's say the user picks wireless. They're then asked:

  • Please open a terminal, type in lspci -nn | grep -i net and copy the output into the following text box. (Help: How to do this - a link to a Q/A on how exactly to follow this procedure)

They paste in the following:

00:0a.0 Ethernet controller [0200]: NVIDIA Corporation MCP79 Ethernet [10de:0ab0] (rev b1)
05:00.0 Ethernet controller [0200]: Atheros Communications Inc. AR242x / AR542x Wireless Network Adapter (PCI-Express) [168c:001c] (rev 01)

We then ask:

  • Which best represents your issue:
    • A problem with your "NVIDIA Corporation MCP79 Ethernet"
    • A problem with your "Atheros Communications Inc. AR242x / AR542x Wireless Network Adapter (PCI-Express)"

They click the Atheros wireless and here's where the magic happens. We do a quick API search for answered questions that contain 168c:001c and offer them up to the user in a "We have questions about this hardware already, please check them to see if they help".

This is where I envisage most people getting off. We have a huge resource of old answered hardware questions and if people knew what to search for, they'd find their answer.

At the bottom of that list we have a "No, none of these helped" link. If the user clicks that, we prepopulate the question with a code-formatted version of the lspci dump (so future lookups can find this version) and any other details we've collected along the way (other examples may ask more questions).

To recap, this has three ridiculously good benefits:

  1. Most people will find their answer before posting and that means less reviewing, less duping.
  2. Those questions that survive don't need follow-up questions because they've already been asked (remember I was suggesting that 40% of questions previously needed human interaction before they could be answerable!)
  3. The unanswerable crap is limited to questions that aren't part of this wizard.

The best thing is this can be built, tested and implemented purely in javascript, using a userscript immediately. It's not a small project but most of the work is in designing thoughtful questions to problems we see a lot of.

  • 7
    Dear GOD, do you know how much I despise hardware troubleshooting questions? And, I am not a big fan of cleaning up after old questions more so than keeping tabs on the currently active questions. I think we should be focusing our efforts on the newest questions really. – jokerdino Aug 15 '13 at 11:17
  • 1
    I agree. The quality of most of them is appalling and that scares people off them. And I agree. Putting all this effort into looking back hurts our ability to look forward. – Oli Aug 15 '13 at 11:33
  • 2
    Please, don't forget about the UEFI-GRUB-dual-boot mess! Hardware is the most difficult to deal, since not everybody has the same hardware, but boot questions are no less in quantity. – Braiam Aug 15 '13 at 12:30
  • 5
    That's exactly the sort of thing a wizard would catch. The first user would click the "My computer doesn't boot" button, then the "I don't see the Ubuntu logo" button and then enter their computer name and then they'll see five questions dealing with their laptop's specific issue. And that will result in them getting their answer and us not getting another duplicate because they didn't know what to search for. – Oli Aug 15 '13 at 12:32
  • Working out these wizards (and what questions to ask) is going to be the hard part. Implementing it over the site is the easy bit. – Oli Aug 15 '13 at 12:33
  • 2
    I don't think a wizard will help, even when we ask people for hw information in comments they bail anyway; I think we'd just get people skipping the dialog anyway. – Jorge Castro Aug 15 '13 at 12:51
  • 5
    Well in fairness it's that sort of user who are cluttering up the site with unanswerable hardware questions. Providing a hurdle for their questions should be a desirable thing. If we can hold their hand before they excrete their malformed post onto the site and actually funnel them to the right place, we've won. If they choose to skip that, that's not our fault - we've tried to help. – Oli Aug 15 '13 at 12:57
  • 2
    I'm sorry, but I am compelled to downvote this answer, because I have strong disagreements to the first three paragraphs or so (before the hardware discussion). While I agree that during a cleanup we don't always handle old questions the right way, that focusing on our "answer" rate is focusing on the wrong thing, and that our rate of answering questions is a concern, the biggest problem (and what I think jokerdino is trying to say) is the quality of the questions coming in. I have completely stopped answering question because I don't have time/the desire to go through all the junk that – Seth Aug 15 '13 at 18:36
  • 2
    comes in, all the question where I have to haggle with the OP for the right information, the questions where the OP abandons and, above all, the questions where I have to explain how to use the site to the OP to get the information I need to answer the question. I just can't do that all the time. I think that every one of jokerdino's propositions would help solve this problem, which would in turn make the rate at which we answer questions go up. – Seth Aug 15 '13 at 18:39
  • Re. the low downvote share, is there a fear that downvoting a question may make the asker go away to some other OS or from Ask Ubuntu? In other words, is there a "conflict of interest" between the need to have more users of Ubuntu versus the standards that SE sites desire to maintain? – user25656 Aug 16 '13 at 3:28
  • @Seth I don't know what you think we disagreed on, but I can't see it. Look at rev 2 (the version you read) and read the paragraph containing "and that's the problem". I have reorganised my argument since then but I haven't changed what I'm saying. – Oli Aug 16 '13 at 9:44
7

We're 2 slots away from being the worst in the entire network:

We need to figure something out, we're two slots away from being the worst. I'm not saying we have to be awesome, but surely we won't be throwing parties over this:

enter image description here

~6%-7% variation from way higher traffic sites, we need to step up!

  • We have reached the bottom. – Seth Dec 24 '13 at 20:43
  • The quality of the product we support has a definite impact on our performance. What I see when I run Number of users answering or questioning Is a massive upsurge in questions around the 13.04 and 13.10 releases, and a reduction around the release of 14.04. I think that as the 2013 versions (which I found too buggy to use seriously personally) age out, things will improve. – Elder Geek May 21 '14 at 15:20
4

Here is my recipe for dealing with the whole "unanswered" thing:

  • I adopted a tag (or more like a set of them) that are more or like what I clean up going from most upvoted and newer questions to far back.
  • I go sometimes to a random page of the Unanswered -> no answers list (ie https://askubuntu.com/unanswered/tagged/?page=20&tab=noanswers) and check up what's there, or just select a tag, and start answering.
  • The normal flagging when I'm doing this.
  • I struggle that most of questions have at least a comment whenever they arrive to the newest page, so new unanswered don't pile up (but I won't do this if I'm busy on other tasks).

You could take some of my behavior, propose some, or just continue what your are doing ;). Although some pre-question quality insurances is always welcomed, since it's tiring having to comment asking for more information about the same things: lspci, lsusb, GRUB-UEFI configuration (the comments vary somewhat, but still is almost the same information that we need), etc.

  • 1
    I think this is a good idea. If we all worked on one tag at a time it would be much better than all of us just flagging whatever unanswered, abandoned questions we came across. It would certainly help lower the mods workload if we all worked together on it. – Seth Aug 15 '13 at 18:52
  • I took the idea from LP package adoption where the triage is done by an adopting parent, then once the report is ready for the developers to fix, they pass em along. The objective is that developers don't extract the problems out of users with questions (like is happening right now). – Braiam Aug 19 '13 at 0:37
4

From the question:

The number of questions we get have significantly increased since 2012 but the answers haven't caught up with them yet. Actually, they are not even trying to catch up.

The increase in the number of questions is partly a natural consequence of the publicity given to Ask Ubuntu over other "help" options. While publicizing Ask Ubuntu, the "ethos" of Stack Exchange is not given prominence and that perhaps accounts for the change in quality. With Ubuntu going mobile, I suspect things will only get worse and the number of people posting in SMSese will rise.

One aspect that hasn't been covered in the question is the number of views per question. I find the low number of views particularly depressing.

Despite what numbers may say, I feel that the community's resources are being spread too thin. Support is being offered

  • at Launchpad
  • via several chat rooms and mailing lists
  • at ubuntuforums.org (despite attempts to depict it as some sort of place for unwanted questions) and
  • at Ask Ubuntu

In addition, someone presumably has to "moderate" the various "social" pages that are mushrooming. The people at Discourse did have a help section but terminated it otherwise there'd be yet another support option. Is the community big enough?

  • 1
    If you see a good question with low views why not just tack a bounty on it or help the guy get more exposure by sharing it on twitter/G+, etc? – Jorge Castro Aug 15 '13 at 12:31
  • @JorgeCastro I did post about the question I cited in the Forums but I get the impression of a house divided. While I'm quite comfortable with both UF and AU, others apparently aren't. Re. the bounty, I suppose I could now but it would be nice to see some sort of commitment from the OP (who is quite active in SO and so should know his way about). Meanwhile, I'll keep hoping for one "mega" unifying source for help on all things Ubuntu rather than the duplication of effort. – user25656 Aug 15 '13 at 13:43
  • 1
    @JorgeCastro If I tacked a bounty on every question that didn't have enough attention I saw I would soon have less than 75 reputation :( Trust me, I want to give bounty's away, but I'd also like to maintain a steady reputation... I like this answer, these things have been bothering me recently (especially after the forums people's attitudes over on discourse), but I haven't been able to express my feelings. This answer does it quite well. – Seth Aug 15 '13 at 18:46
  • In order to accommodate Ubuntu's wide user base, it is necessary to spread support efforts over several different support systems. Most significantly, many Ubuntu users don't speak English, and Ask Ubuntu is an English-only site. Some users prefer to minimize or eliminate the use of proprietary software (including webapps beyond a certain level of complexity), which should be respected but means they can't participate actively on AU. Some people have questions that need so much back-and-forth interaction (or want intensive help reporting bugs): they need a forum. And SE chat requires rep. – Eliah Kagan Aug 16 '13 at 8:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .