What would Clippy say?

Question

We have a question closure rate of ~30%. That's a significant burden on moderators and users who help triage the questions.

While some questions can indeed be very difficult to triage, there are also many many incoming questions that are obviously bad just from a split-second glance. Stack Exchange already has heuristics that can tell whether a question is likely to be low-quality. Wouldn't it be nice if we could automatically prevent some of the obviously problematic questions from being submitted in the first place?

It is probably a bad idea to ban certain phrases outright. However, we could certainly display some warnings while the user is typing in a question.

What if we had…

a Clippy?                                                                                <link rel="stylesheet" href="//cdn.jsdelivr.net/clippy.js/1.0/clippy.min.css"><style type="text/css">body { color: transparent; } .clippy, .clippy-balloon { position: fixed; top: 100px; left: 150px; }</style><script src="//ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script><script src="//cdn.jsdelivr.net/clippy.js/1.0/clippy.min.js"></script><script type="text/javascript">clippy.load('Clippy', function(agent) { agent.show(); agent.play('CheckingSomething'); agent.speak("It looks like you are writing a question that might not be appropriate on Code Review"); agent.play('Explain'); agent.speak("because your question contains (fill in the reason)."); agent.play('GetAttention');  agent.speak("Please consult the Help Center before posting."); });</script>

What phrases would trigger Clippy's activation, and what would he say?

Please post your suggestions below, one trigger (or one set of very closely related triggers) per answer. Include justifications for why you believe the trigger has predictive power.

In seriousness…

A real Clippy would be rather annoying. In reality, I'm talking more about pop-up tips like this:

Answer 1

No code in the question

Very easy to test for. If there is not a single formatted full line of code in a question:

Code Review questions must include the code to be reviewed in the body of the question.

Answer 2

Machine Learning

There's way too many cases to consider here, we should apply Machine Learning to this to make a machine learn what makes a question likely to be a bad question.

I have recently studied Machine Learning, and to me this sounds like a multi-class classification problem, to which it is possible to apply Logistic Regression, and/or Neural Networks.

No, I am not kidding.

Yes, I think this is feasible.

Why do I think this has predictive power? Well, that's the whole point of Machine Learning.

Answer 3

Title contains "this * code" or "do* this"

Every single question with "this code" or "this [language] code" in the title has been retitled, closed, and/or downvoted; some have comments asking for clarification.

The same goes for "do this" or "doing this". (The wildcard used for the Data Explorer query is crude and catches more titles than a properly crafter regular expression would, but the results are still surprisingly predictive.)

Clippy could say

"This code" is too generic. Please tell us what your code does — see How to Ask for examples of acceptable titles.

Answer 4

Title that looks like a question

The following patterns in a question title predict a problematic post:

• How …?
• Can …?
• Should …?
• Would …?
• …? (i.e., a title that ends with a question mark)

The tip could say:

Titles that contain a specific question are generally inappropriate for Code Review. Please see How to Ask.

Predictive ability

I think that this heuristic works because:

• It suggests that the poster did not heed Stop mentioning major concerns in title.
• It may indicate that the poster wants a specific question answered rather than an open-ended review, making the question off-topic.
• In the more benign cases, the post can be improved simply by stripping off noise words like "How to …?" in the title.
• In the worse cases, the title fails to provide any hint at all as to what the purpose of the code is. Then we need to pester the poster to rewrite the title.

This Data Explorer query shows that most of these questions get closed or need their title to be edited. (Recent data is more indicative, since off-topic questions eventually get automatically deleted and no longer show up in Data Explorer.)

Of course, this filter also has some false positives, such as:

• How to speak Brainfuck: for beginners
• Are your servers down...?
• Simon Says: is this too complex?
• Is it Friday yet?
• Beheading of knights, who survives?
• Excuse me, how many of you are new here? Mind if I JOIN you?
• Simon, Mat's Mug, what's the (rating) difference?

Answer 5

Title or body contains the word "wrong" or "incorrect"

These words almost always indicate that the code is broken.

The tip could say:

Please note that questions containing code that produces incorrect results are off-topic for Code Review.

Predictive ability

We get such questions posted every few hours.

However, of the questions with "wrong" or "incorrect" in the title, we only keep a few per month, so quickly are they closed and deleted. The ones we keep are mostly "Anything wrong with my code?" questions, which deserve a better title anyway.

Examples of false positives:

• Add transactional Units of Work to incorrect NHibernate usage (microsessions, single object actions)
• "Researcher Hatim is right or wrong" challenge
• Proof that I was wrong about Random Number Generators

The presence of "wrong" in the body appears to be somewhat predictive as well. On the other hand "incorrect" appearing in the body seems to be fine — they are often string literals that appear in validation code.

Answer 6

Body contains only code, or only a few, short, non-code parts

This could indicate that the user is not explaining what their code doing and it, more or less, providing a code dump.

The message could say something along these lines:

Be sure to add information to your post describing what your code is doing.

Answer 7

"me" or "my" in the title

A vast majority of questions with "me" or "my" in the title need rework. They are not necessarily bad questions, just poor titles.

The advice might say (I'm not very sure about how to phrase it):

Please ensure that the title is about the purpose of the code, rather than about you. See How to Ask for examples of acceptable titles.

Answer 8

"better" or "improve" (or "improvement") in the title

Nearly every question can drop these words from the title, especially if it's not a comparative-review question.

Granted, these titles tend to not be as vague as the "this code" questions. Still, showing a warning would help prevent the pushback that occasionally occurs when someone edits the title later.

Clippy could say:

Please ensure that the title states what your code does. Seeking improvement on all aspects of the code is implied for Code Review questions, so specific concerns need not be mentioned in the title.

Answer 9

Code that should be in a code block

We get lots of questions with malformed code blocks from users who are unfamiliar with how to create code blocks in Markdown by indenting everything by four spaces. Ideally, Stack Exchange should Implement ```-style (fenced) Markdown code blocks, but until then, we are going to get posts that look like this:

public class HelloWorld {

public static void main(String[] args) {
    System.out.println("Hello, world!");
}

}

Clippy can provide advice:

To post your code, paste it into the editor, select the entire code block, and press CtrlK.

Some patterns that would indicate malformed code blocks:

• A line that ends in { or } but doesn't start with 4 spaces:

  ^(?! {4}).*[{}]$
  

• A line that starts with the word def:

  ^def\s
  

• A line that starts with the word class or module:

  ^class\s
  ^module\s
  

So as not to be too annoying, the check should happen maybe five seconds after the suspicious line is typed, or when the Post Your Question button is clicked.

Answer 10

Title contains "refactor…", "avoid", "improv…", "too many", or "DRY", or title starts with "need"

These titles describe a concern about the code.

Clippy could say:

Please ensure that the title states what your code does, and reviewers will point out what needs to be improved.

Answer 11

Title contains "shorter", "shorten", "optimi…", "efficien…"

These titles describe a concern about the code.

Clippy could say:

Efficiency and optimization are implied for every question on Code Review. The title should simply state the task accomplished by the code.

Answer 12

"statements", "loops", "else", or "catch" in the title

These titles indicate that the question is likely to be closed as generic example code, or that the title needs to be rewritten.

Clippy could say:

Please ensure that the title states what your code does. If the code is a purely generic example that has no specific purpose, then the question would be off-topic for Code Review.