43
\$\begingroup\$

We have a question closure rate of ~30%. That's a significant burden on moderators and users who help triage the questions.

While some questions can indeed be very difficult to triage, there are also many many incoming questions that are obviously bad just from a split-second glance. Stack Exchange already has heuristics that can tell whether a question is likely to be low-quality. Wouldn't it be nice if we could automatically prevent some of the obviously problematic questions from being submitted in the first place?

It is probably a bad idea to ban certain phrases outright. However, we could certainly display some warnings while the user is typing in a question.

What if we had…

a Clippy?                                                                                <link rel="stylesheet" href="//cdn.jsdelivr.net/clippy.js/1.0/clippy.min.css"><style type="text/css">body { color: transparent; } .clippy, .clippy-balloon { position: fixed; top: 100px; left: 150px; }</style><script src="//ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script><script src="//cdn.jsdelivr.net/clippy.js/1.0/clippy.min.js"></script><script type="text/javascript">clippy.load('Clippy', function(agent) { agent.show(); agent.play('CheckingSomething'); agent.speak("It looks like you are writing a question that might not be appropriate on Code Review"); agent.play('Explain'); agent.speak("because your question contains (fill in the reason)."); agent.play('GetAttention');  agent.speak("Please consult the Help Center before posting."); });</script>

What phrases would trigger Clippy's activation, and what would he say?

Please post your suggestions below, one trigger (or one set of very closely related triggers) per answer. Include justifications for why you believe the trigger has predictive power.

In seriousness…

A real Clippy would be rather annoying. In reality, I'm talking more about pop-up tips like this:

"The question you're asking appears subjective and is likely to be closed"

\$\endgroup\$

12 Answers 12

44
\$\begingroup\$

No code in the question

Very easy to test for. If there is not a single formatted full line of code in a question:

Code Review questions must include the code to be reviewed in the body of the question.

\$\endgroup\$
  • 10
    \$\begingroup\$ This is more of a hard error when clicking submit than a Clippy message while typing, but I'll take it. \$\endgroup\$ – 200_success Mar 30 '16 at 8:07
  • 1
    \$\begingroup\$ Maybe a warning if somebody embeds a link to github/pastebin/(other common code hosting service)? \$\endgroup\$ – Kaz Mar 30 '16 at 8:09
  • 2
    \$\begingroup\$ There's nothing wrong with posting a link to your project IMO. Say you post a class for review, it could be helpful to provide a github link as a compliment. \$\endgroup\$ – jacwah Mar 30 '16 at 14:33
  • 1
    \$\begingroup\$ Linking to a repository is just a potential warning sign. Something along the lines of "Please make sure that you have included the code you want reviewed in the question". \$\endgroup\$ – Kaz Mar 30 '16 at 14:44
  • 1
    \$\begingroup\$ @Kaz I for one always include all the relevant code in the question and link to Git because if there are dependencies as submodules, people'd rather clone \$\endgroup\$ – cat Apr 1 '16 at 0:55
  • 2
    \$\begingroup\$ Ok, done: codereview.meta.stackexchange.com/questions/9038/… \$\endgroup\$ – Shog9 Dec 12 '18 at 4:39
29
\$\begingroup\$

Machine Learning

There's way too many cases to consider here, we should apply Machine Learning to this to make a machine learn what makes a question likely to be a bad question.

I have recently studied Machine Learning, and to me this sounds like a multi-class classification problem, to which it is possible to apply Logistic Regression, and/or Neural Networks.

No, I am not kidding.

Yes, I think this is feasible.

Why do I think this has predictive power? Well, that's the whole point of Machine Learning.

\$\endgroup\$
  • 6
    \$\begingroup\$ I like this suggestion much better than trying to guess some poor heuristics by hand. That said, since it would be a major undertaking to implement, it won't happen from alone. \$\endgroup\$ – 5gon12eder Mar 30 '16 at 14:30
  • \$\begingroup\$ Can a community implement this by themselves? Shouldn't this rather be posted as a feature request on the general SE meta? I can see this being useful on all communities. \$\endgroup\$ – jacwah Mar 30 '16 at 14:35
  • 9
    \$\begingroup\$ I had considered piping everything to SpamAssassin and training it. \$\endgroup\$ – 200_success Mar 30 '16 at 15:07
  • 4
    \$\begingroup\$ @jacwah The rules are going to be very different on each community though. For example, here a question mark is a sign of a bad title. On other sites, its lack would be. Also, there are a lot of advantages to trying to do this in a small way initially rather than a big way. I would hate to be stuck with code that did not work for this site but which would be a low priority to fix since it worked everywhere else. \$\endgroup\$ – mdfst13 Mar 30 '16 at 17:34
  • \$\begingroup\$ @mdfst13 I wasnt refering to this question as a whole but this answer in particular. Implementing a machine learning algorithm would be a general solution applicable to all communities. It could just be run as a separate instance for each one. \$\endgroup\$ – jacwah Mar 30 '16 at 18:40
  • 2
    \$\begingroup\$ @jacwah In response to "Can a community implement this..." I agree; this could potentially be helpful across all of SE. However, I fear that if we put this on the general SE meta, it will take 6 - 8 weeks before they start working on it. I'd say it'd be fine in the mean time if we tried implementing this ourselves. \$\endgroup\$ – SirPython Mar 30 '16 at 21:38
  • \$\begingroup\$ Would Duga take control of this, and perhaps print her findings to the 2nd? \$\endgroup\$ – SirPython Mar 30 '16 at 21:39
  • \$\begingroup\$ @SirPython That makes sense. \$\endgroup\$ – jacwah Mar 30 '16 at 22:31
  • 1
    \$\begingroup\$ @jacwah Any developer who is familiar with Machine Learning can implement this by themselves. \$\endgroup\$ – Simon Forsberg Mar 31 '16 at 7:17
  • \$\begingroup\$ Just use this: cs.stanford.edu/people/karpathy/convnetjs JS neural network \$\endgroup\$ – noɥʇʎԀʎzɐɹƆ Apr 2 '16 at 0:06
  • \$\begingroup\$ @CrazyPython The neural network library is not the hard part, deciding on the features, gathering data and analyzing results is a harder part of Machine Learning. \$\endgroup\$ – Simon Forsberg Apr 2 '16 at 12:44
  • \$\begingroup\$ Ping me when you start the project. I'll be happy to contribute. \$\endgroup\$ – janos Apr 3 '16 at 16:31
  • 1
    \$\begingroup\$ Well, this is one of the most massive programming communities on SE. I think we have the right dev community to give it a try at least :) \$\endgroup\$ – BrainFRZ Apr 13 '16 at 3:21
  • 2
    \$\begingroup\$ Questions on SE are closed for various reasons. Training a ML model that can handle all cases for all different sites together is going to be hard. Tackling specific scenarios and building it up from there seems a good way to start. As @SimonForsberg said, anyone can use a ML library and just look at how to use its API. It's however important to select the correct features that match the scenario. E.g. One of the features could be the presence of code or absence, or the length of code snippets. Still machine learning is no silver bullet, there will be false positives. \$\endgroup\$ – DJanssens Apr 14 '16 at 15:16
  • 2
    \$\begingroup\$ We use something like this on Stack Overflow. It's slightly accurate on a good day, and requires a lot of handholding to be useful. But if you're motivated to try doing something for CR, download the data-dump and give it a shot... \$\endgroup\$ – Shog9 Apr 20 '16 at 17:38
26
\$\begingroup\$

Title contains "this * code" or "do* this"

Every single question with "this code" or "this [language] code" in the title has been retitled, closed, and/or downvoted; some have comments asking for clarification.

The same goes for "do this" or "doing this". (The wildcard used for the Data Explorer query is crude and catches more titles than a properly crafter regular expression would, but the results are still surprisingly predictive.)

Clippy could say

"This code" is too generic. Please tell us what your code does — see How to Ask for examples of acceptable titles.

\$\endgroup\$
  • 1
    \$\begingroup\$ Very useful query. I can't say I'm surprised by the number of bad titles, but at least this makes it easier to fix them. \$\endgroup\$ – Jamal Mar 30 '16 at 23:53
23
\$\begingroup\$

Title that looks like a question

The following patterns in a question title predict a problematic post:

  • How …?
  • Can …?
  • Should …?
  • Would …?
  • …? (i.e., a title that ends with a question mark)

The tip could say:

Titles that contain a specific question are generally inappropriate for Code Review. Please see How to Ask.

Predictive ability

I think that this heuristic works because:

  • It suggests that the poster did not heed Stop mentioning major concerns in title.
  • It may indicate that the poster wants a specific question answered rather than an open-ended review, making the question off-topic.
  • In the more benign cases, the post can be improved simply by stripping off noise words like "How to …?" in the title.
  • In the worse cases, the title fails to provide any hint at all as to what the purpose of the code is. Then we need to pester the poster to rewrite the title.

This Data Explorer query shows that most of these questions get closed or need their title to be edited. (Recent data is more indicative, since off-topic questions eventually get automatically deleted and no longer show up in Data Explorer.)

Of course, this filter also has some false positives, such as:

\$\endgroup\$
  • 2
    \$\begingroup\$ ... so you hit the "ask question" button, but if you then have a question mark then it's a bad question? Your data point might be valid, but there's something else wrong about it - maybe on CR we aren't looking for questions ... but rephrasing that is something that would be pretty hard for SE to do, I think. \$\endgroup\$ – Pimgd Apr 4 '16 at 12:24
19
\$\begingroup\$

Title or body contains the word "wrong" or "incorrect"

These words almost always indicate that the code is broken.

The tip could say:

Please note that questions containing code that produces incorrect results are off-topic for Code Review.

Predictive ability

We get such questions posted every few hours.

However, of the questions with "wrong" or "incorrect" in the title, we only keep a few per month, so quickly are they closed and deleted. The ones we keep are mostly "Anything wrong with my code?" questions, which deserve a better title anyway.

Examples of false positives:

The presence of "wrong" in the body appears to be somewhat predictive as well. On the other hand "incorrect" appearing in the body seems to be fine — they are often string literals that appear in validation code.

\$\endgroup\$
  • 2
    \$\begingroup\$ Further keywords might include "Stacktrace", "Error", "Fix", but those may have too many false-positives \$\endgroup\$ – Vogel612 Mar 30 '16 at 8:08
  • \$\begingroup\$ Don't forget "Segfault" and the likes. False positives aren't as problematic when warning IMO. \$\endgroup\$ – Mast Mar 30 '16 at 10:57
  • \$\begingroup\$ This could potentially trigger a lot of false positives. Some questions describe a (supposedly) clean way they had originally tried, but it didn't work, so they figured it out, and want help cleaning up their code. \$\endgroup\$ – 23fc9a62-56de-47fb-97b4-737890 Mar 30 '16 at 15:44
  • 1
    \$\begingroup\$ for this kind of questions it would help to have stats on how many of their askers simply try to circumvent question block at SO (here is example stats request at Progs meta just in case if you don't have these). Thing is, for this kind of askers heuristics and Clippy message could probably be different - that is, in case if stats show that there are many of them \$\endgroup\$ – gnat Mar 30 '16 at 16:23
  • 1
    \$\begingroup\$ @gnat I'm not confident enough to block anything at this point. The goal is purely advisory, and therefore circumvention is not a concern. The crappiest posters gonna post crap. \$\endgroup\$ – 200_success Mar 30 '16 at 16:26
  • \$\begingroup\$ I'm not talking about block, only better targeted warning "Wait... Note that Stack Overflow topics are not covered here and that your questions can't even be migrated to Stack Overflow because of the block." But even this is moot since without stats there is no way to tell whether it's worth it, what I really recommend is to request stats and act based on these \$\endgroup\$ – gnat Mar 30 '16 at 16:30
  • \$\begingroup\$ ...(on a further thinking) I would go as far as to say that blocking askers only by the fact of block at SO would be really bad idea. Their inability to properly ask SO kind questions can't reliably indicate inability to ask CR kind ones - sites just differ too much for that. All these folks need is indeed a fair warning that SO kind troubleshooting is off-topic at CR and that their questions can't migrate to SO because of the block \$\endgroup\$ – gnat Mar 30 '16 at 21:28
19
\$\begingroup\$

Body contains only code, or only a few, short, non-code parts

This could indicate that the user is not explaining what their code doing and it, more or less, providing a code dump.

The message could say something along these lines:

Be sure to add information to your post describing what your code is doing.

\$\endgroup\$
  • 1
    \$\begingroup\$ off my head: [code that does something] // this does XYZ, but slowly, how to improve? :-) \$\endgroup\$ – LIttle Ancient Forest Kami Apr 14 '16 at 21:18
13
\$\begingroup\$

"me" or "my" in the title

A vast majority of questions with "me" or "my" in the title need rework. They are not necessarily bad questions, just poor titles.

The advice might say (I'm not very sure about how to phrase it):

Please ensure that the title is about the purpose of the code, rather than about you. See How to Ask for examples of acceptable titles.

\$\endgroup\$
  • \$\begingroup\$ Seems this one should be combined with the "this * code" filter. \$\endgroup\$ – Mathieu Guindon Apr 12 '16 at 0:26
9
\$\begingroup\$

"better" or "improve" (or "improvement") in the title

Nearly every question can drop these words from the title, especially if it's not a question.

Granted, these titles tend to not be as vague as the "this code" questions. Still, showing a warning would help prevent the pushback that occasionally occurs when someone edits the title later.

Clippy could say:

Please ensure that the title states what your code does. Seeking improvement on all aspects of the code is implied for Code Review questions, so specific concerns need not be mentioned in the title.

\$\endgroup\$
6
\$\begingroup\$

Code that should be in a code block

We get lots of questions with malformed code blocks from users who are unfamiliar with how to create code blocks in Markdown by indenting everything by four spaces. Ideally, Stack Exchange should Implement ```-style (fenced) Markdown code blocks, but until then, we are going to get posts that look like this:

public class HelloWorld {

public static void main(String[] args) {
    System.out.println("Hello, world!");
}

}

Clippy can provide advice:

To post your code, paste it into the editor, select the entire code block, and press CtrlK.

Some patterns that would indicate malformed code blocks:

  • A line that ends in { or } but doesn't start with 4 spaces:

    ^(?! {4}).*[{}]$
    
  • A line that starts with the word def:

    ^def\s
    
  • A line that starts with the word class or module:

    ^class\s
    ^module\s
    

So as not to be too annoying, the check should happen maybe five seconds after the suspicious line is typed, or when the Post Your Question button is clicked.

\$\endgroup\$
  • \$\begingroup\$ Would that regex have to parse the HTML source inside the preview? If so, we could perhaps just look for a curly brace anywhere between <p></p> tags (see the HTML source for your malformed code block) \$\endgroup\$ – Phrancis May 9 '16 at 18:11
  • \$\begingroup\$ @Phrancis Good question. I don't know whether Clippy would receive the Markdown or the rendered HTML. I've assumed Markdown here. \$\endgroup\$ – 200_success May 9 '16 at 18:13
  • \$\begingroup\$ Okay, I'm thinking of @Duga for reference, she uses the HTML to parse the edits for answer invalidation, if any edited text is between <code> tags, not sure if Clippy would use the same method though, guess that would be more of an implementation detail though. \$\endgroup\$ – Phrancis May 9 '16 at 18:16
5
\$\begingroup\$

Title contains "refactor…", "avoid", "improv…", "too many", or "DRY", or title starts with "need"

These titles describe a concern about the code.

Clippy could say:

Please ensure that the title states what your code does, and reviewers will point out what needs to be improved.

\$\endgroup\$
5
\$\begingroup\$

Title contains "shorter", "shorten", "optimi…", "efficien…"

These titles describe a concern about the code.

Clippy could say:

Efficiency and optimization are implied for every question on Code Review. The title should simply state the task accomplished by the code.

\$\endgroup\$
4
\$\begingroup\$

"statements", "loops", "else", or "catch" in the title

These titles indicate that the question is likely to be closed as generic example code, or that the title needs to be rewritten.

Clippy could say:

Please ensure that the title states what your code does. If the code is a purely generic example that has no specific purpose, then the question would be off-topic for Code Review.

\$\endgroup\$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .