Does Code Review systematically prefer the least important code, and what can we do about it?

Question

There are already a large number of intrinsic hurdles before production code can get reviewed; it can be against company policy, paid developers are likely less open to criticisms and new coders are the only ones actually actively looking for help.

But then Code Review comes to stomp on the little fire we have:

• Code can't really be anonymized (else it's "theoretical"),
• Good code doesn't get reviewed as much,
• Complex code gets ignored,
• Votes further award "lowest common denominator" questions,
• Code must be "snippeted", which makes extracts from production code unrunnable.

Even the low volume of commentary on questions is likely to contribute - newbies often get immediate, simple answers (naming! comments!) but no such activity exists for most professional questions.

I consider "toy questions" to be anything where the code is not intended to be used, or doesn't try to do something of value. To be clear, it's not that I don't accept the toy questions or challenges. However, there is a downward spiral, in that the prevalence of exclusively toy questions enforces the notion that Code Review is for toy questions.

I don't know how we could change this, but it seems one would need a dedicated call to giving attention and good-quality answers to professional code. We should at least keep up the impression - like in-person peer review, the emphasis should be on immediate and hypothetical suggestions, not ground-up rewrites. Voters should appreciate this, and not leave such answers ignored. Perhaps even bounties could get involved - who knows?

(Let's not forget that "looks good" after a few clarifying questions is a valid review, but Code Review has almost no such answers.)

If we can't shake of this stigma now, we may never be able to.

Do you agree, or have ideas on how to fix it?

Answer 1

I want to address the time-constraint issues. Time has become increasingly precious to me, and an indepth review of complicated code can take a lot of time.... there are answers where I have spent days pondering the review. That sort of time commitment is unreasonable for a site like this.

In fairness, I volunteer the time, I learn things as I go, and it's a mutually beneficial process, but the process is neither scalable, nor sustainable.

On a personal level, I like the small quick & dirty reviews because I like helping people with immediate lessons on how they can improve things - If I have a "this is my 2 cents" review on hand, I am quick to give it.

On the other hand, I also enjoy the big and involved reviews when I have done them, but that is more of a mentoring than reviewing thing. I just can't do that many of them.

I think bounties is an option. I think that a separately maintained list of "candidate" questions (in meta?) is a possibility. I would sponsor bounties for anyone who adequately answered one of the "big hitters"....

Consider something like this:

1. if a question is very high quality, has a professional nature to it, is complex, comprehensive, or whatever, post a meta-question linking back to it with a special tag, say: promoted-question.
2. People can "answer" the meta post with offers to answer, or contribute a bounty.
3. When answers come in, the meta-question can be marked as "complete" (and bounties, if offered, can be awarded/debated).

People who are interested in significant investments of time, or have such availability, can then use meta to both identify, discuss, and track these high-quality questions.

If such a system (or something similar - even if it is not on meta) were established I would be willing to seed the process with promises for bounties - say 250 rep minimum for the first 10K worth of professional grade questions - until the process becomes self-sustaining.

Even if the system above is too complicated, or takes a while to implement, please ping me if you find a question that you feel is a high quality question that is under-exposed. I will, on a case by case basis, volunteer bounties on those questions.

Answer 2

"Looks good" is a really hard answer to post. You have to be extraordinarily confident in order to be able to post that. Moreover, if a question is truly up to the standards of Code Review and we really want "Looks good" answers, then every question should have an immediate self-answered "Looks good" answer from the asker. If the asker can't say "Looks good", then the code likely isn't really ready for a review.

_{I know this isn't a complete answer to the question asked. I just want to address this point specifically.}

Answer 3

I think I have a similar feeling, but it's hard to tell, because your question is full of vague or inaccurate elements. I try to address your and main remarks, but if I misunderstood your main underlying objections then please forgive me.

• Does Code Review systematically prefer the least important code

  • Maybe you meant easy code? As in, easy to answer questions with simple answers get a lot of votes, or get hot? In my experience on multiple SE sites (CR, SO, UNIX), posts that are easy to understand by a broad audience are more popular, get more attention and more votes. This is inevitable.

• Code can't really be anonymized (else it's "theoretical")

  • I'm going to assume that you're talking about posting proprietary code, from work. I haven't had to pleasure yet to work at a company that would allow posting our code, even anonymized. If it's not proprietary code, then you don't need to anonymize. So this point seems moot.

• Good code doesn't get reviewed as much

  • Doesn't it? I've seen many reviews of good code, and done many reviews of good code myself. I often take pleasure in reading good code, and often close the tab when I don't like what I'm seeing. I haven't noticed the tendency you describe.

• Complex code gets ignored

  • It's not that it gets ignored, it just has limited audience. See my first remark about easy questions.

• Votes further award "lowest common denominator" questions

  • Again, this remark seems to be about easy questions. Slightly more clear than saying "least important questions", but this is still a vague statement.

• Code must be "snippeted", which makes extracts from production code unrunnable.

  • I don't see how code "must be" snippeted. Granted, short code will tend to get more reviews than long code, as it falls in the easy question category I described above. It's only natural that longer code takes more effort to review than shorter code, so it's also natural that you have to work on it harder.

Perhaps I can reframe your question, and ask some additional questions:

• What can we do to drive more attention to harder questions?

  • The OP can do the most, by making the question more interesting. We've done something for that, by meta posts about how to ask good questions. But in the end, this comes done to the OP.
  • I like @rolfl's idea of promoted-questions

  • Change the newsletter content: looking at the last one, it includes:

    • 7 highly popular questions
    • 2 from past popular questions
    • 3 unanswered popular questions

    Perhaps we can include more unanswered questions. Perhaps we can highlight bounties too.

Answer 4

Who determines what's important?

Does Code Review systematically prefer the least important code, and what can we do about it?

Yesterday I asked two questions, one was a follow-on from the first. The concepts in the question are very new to me because I am a beginner in the languages/tooling that's used, but, to be clear, the concepts involved, and perhaps even most of the code itself, will become part of a much larger, real, and "professional" system.

To me, the code was very, very important. The answers, and subsequent discussions have, for the most part, been very helpful, and corrected some basic misunderstandings I have, and have improved my "quality" as a JavaScript/node.js programmer.

While the questions in one hand could be considered "toy" questions, on the other hand, they are pivotal to my personal understanding.

The answers to those questions have already proven to be immensely valuable. The questions are important.

Let's not forget that all questions are important to the person who asks them, regardless of how 'professional' or 'perfect' the code may be.

Just because we struggle to handle "big" and "complex" questions does not mean we are doing the wrong thing on the smaller, simpler questions.

Code Review is doing the right thing on small questions, and we should not be trying to change that (at least not in a bad way).

Answer 5

Maybe we need other kinds of incentives...

Joke aside, this is one question where Mat's Mug's answer is quite apt. It's not that complex code are systematically ignored, they simply take more time and effort to triage/diagnose/dissect.

Also, based on my personal experience, it's generally easier to infer some form of context for shorter questions when they are lacking, which is good enough to provide a basic answer while the question may undergo further clarifications/edits. It's not as easy if one faces a wall of code with little context.

What I'm trying to say here is when longer questions provide little context or background, it is also much harder to provide an appropriate, thorough answer as expected.

Answer 6

One thing that irks me is that unless there's a significant variance in quality, generally the most upvoted answer is one that was posted early, as it gets the most views. If upvoting is simply recognition of a contribution of value, this makes sense, but after a few days when you come back to a question the number upvotes don't necessarily reflect the quality of the answers.

If a top-quality answer is posted after the initial wave of front-page views and upvotes, it will still appear second to earlier answers unless it goes on a run to come from behind, which means that either people who upvoted the first answer come back and upvote the new answer as well, or users viewing the question for the first time only upvote the new answer, which generally means that new answer had better blow the other ones out of the water.

I believe the rep incentives of the current system unintentionally reward quick answers and contribute towards an inverse proportionality between upvotes and quality answers.

I'm thinking Code Review might benefit from something like Google's awarding ad space to the second-highest bidder. What if reviews could be submitted but not seen or upvoted by anyone but the asker for a while, and reviewers get alerted back to the question when that time period is over?

I just came up with that idea as an example and there are some obvious flaws with it, but I think we might benefit from mixing up the process in a similar fashion.