Should comparisons of code snippets be closed as "not seeking a code review"?

Question

I've been seeing some "A vs. B" questions for a while, and I'm not sure if they're considered on-topic for this site. Many of these questions are not asking for a review, but which code snippet is better. These, to me, sound primarily opinion-based.

However, I've seen some that are asking about best practices rather than just straight opinions. I wouldn't like to clump all these questions together, and I still think each question should be evaluated to make sure they can still be reviewed.

Here's one "A vs. B" question that has more to it than just asking for opinions:

Nesting versus GOTO: which is better to avoid?

It's clearly not just asking, "Here's A and B; which is better?" It is, first and foremost, real working code written by the OP. It also does not read like a poll, which is a good indication that the OP has put effort into this code and wants to learn from it. I personally would continue to allow questions like these.

Essentially, I want to know if we should allow "A vs. B" questions that are otherwise on-topic for this site (working, non-hypothetical, and belongs to the OP). I understand that many people find these questions popular, but looking for signs of off-topicness should come first, even if it becomes a hot question.

Answer 1

The mission of Code Review is to help programmers improve working code. Although A-B comparison questions tend to elicit a different kind of answer and therefore make us feel uncomfortable, they are just another way for programmers to seek help to improve their working code. We should therefore welcome them as Code Review questions, even though they don't seek a "traditional" review.

Let's assume that the A-B comparison contains two code excerpts that meet our usual criteria (i.e., non-hypothetical working code from a project that you own or maintain). Otherwise, we would just consider the question to be off-topic on those grounds.

Answers to A-B comparison questions would probably take one of the following forms:

1. A (or B) is better, because (reason).
2. A and B each has its drawbacks: (analysis). Pick your poison.
3. Hey, did you notice a bug in A? Also, B could be better written as (code).
4. A and B both suck. Do C instead, which is better because (reason).
5. Neither one is significantly better or worse than the other. Which one you pick is a matter of personal preference.

Most of those are beneficial answers, and it would be a shame to close A-B comparison questions in general. There's no obvious way to predict which kind of answer might come out of an A-B comparison question. After all, if you already knew, then you wouldn't need to ask in the first place. Only Type 5 has a possibility of being unproductive; you can downvote poor quality questions without closing them, or close the really lame ones as "Primarily opinion-based".

@JeffGohlke's Nesting vs. Goto question managed to produce excellent answers in several of those categories. The question was skillfully asked, but even if it had consisted of a terse "Which is better — A or B?" and two code dumps, it might still have elicited good answers.

We already have many "Which is better?" questions, so obviously the demand exists. Each question represents a desire from a fellow programmer who wants to improve their code. Let's focus more on helping, and less on whether the question was phrased in a way that suits our tastes.

Answer 2

I'm the author of that particular question, and I just wanted to chime in with my experiences with it. (Sort of a ground-floor perspective, if you will.)

Yes, the question as asked was about comparing two particular styles to figure out which was better, which was the lesser of two evils. But the fact was that I had been staring at that particular piece of code for so long that I had tunnel-vision and had lost sight of the forest through the trees. The "correct" answer to that question (as much as there are objectively correct answers at all on Code Review) was neither option A nor option B -- it was options C through F.

Had I not posted the details of my use-case and the actual working code I was dealing with, I agree that it ought to have been closed. But because I included a good bit of the code I had written, the answerers were able to approach it with a fresh perspective and say, basically, "You have a false dichotomy here. You don't have to choose between these two bad things. There are lots of other ways to tackle the problem."

As the asker, I found those answers very helpful because they made me take a step back, breathe, and take a long, serious look at my problem. They reminded me of some core tenets which often come up in code reviews, such as "extract 'til you drop" and "don't reinvent the wheel". And in the end, these made the final version of my code much better and will remind me in the future to back off a bit and look at alternatives which are outside the narrow constraints of a few particular lines. If the point of a code review is to improve a piece of code and to improve the coder's abilities, then it was a successful code review.

As a secondary or even tertiary benefit, I've found that questions which are phrased as such (generalized with wide applicability; common programming conundrums which everyone faces at some point, across many/all languages) are the ones which give this Stack Exchange site the most visibility via the Hot Questions list. Sometimes, due to a sporadic influx of votes, a question will make it up there for 15 minutes, but this is usually the best case scenario for Code Review. On the other hand, questions like the one you link to tend to linger on the Hot Questions link for 24 to 72 hours. I realize that site visibility shouldn't be a criterion for whether or not a question is "on topic", but it's a benefit which shouldn't be entirely ignored, especially given the concern of drawing in new users and retaining them.

Answer 3

While I don't think we should completely remove comparative-review, I do think it might be a good idea to considerably narrow the scope on what's allowed under this tag.

---

Comparative reviews should be written in the same language.

While it's not impossible to be sitting on the fence about which language to write your code in, if you're at this point, you're not ready for CodeReview. Moreover, if you're posting this question, you're sort of really asking, "which language is better, A or B?" and I don't think this is really all that great of a question for Code Review. Even if you're writing an Objective-C/Swift application in which the languages can be interchanged on a file-by-file basis, you still need to be decided on which language you're writing in before you're actually ready for a review. The choice of which language to use is outside the scope of Code Review.

---

Comparative reviews should not be performance or optimization reviews.

If you're trying to optimize execution speed, determine which of your two snippets runs faster and post that snippet by itself as a speed-optimization question.

If you're trying to minimize your applications memory footprint, determine which of your two snippets has a smaller memory footprint and post that snippet by itself as a memory-optimization question.

If you don't know how to determine which is better from a well-defined metric based standpoint, then search StackOverflow for how to measure whatever metric you're trying to compare. There are answered questions on how to make this comparison in basically every language.

---

Comparative reviews should not be too broad.

While it's okay to post relatively broad questions on Code Review along the lines of "This is my implementation of [x]. What can be improved regarding readability? Am I following best practices?", a question this broad does not work that well for a comparative-review question.

What, specifically, are you looking to compare between the two snippets? What do you believe to be the pros and cons of the two snippets, comparatively? And what, specifically, are you looking for in terms of determining which snippet is better?

Give reviews some idea of the criteria they should use in order to recommend to you the better snippet.

---

Comparative reviews might be better asked as two separate questions.

There's nothing preventing you from asking two questions.

If you're still trying to decide on a language, write one implementation in language A and post that question, and language A experts will answer with any and all improvements they see that can be made. While that question is collecting answers, write and implement the same code in language B and post that as its own question. Language B experts will answer that question with any and all improvements. Collect all the improvements for both implementations, implement them all, and make the comparisons for yourself--this may help you decide on a language.

If you're trying to create an optimized implementation, post both of your current implementations as starting points. See what sort of optimization answers each question collects. Implement these optimizations, then compare the updated versions of the code yourself.

Even a comparative review such as "which snippet is the cleanest" is still probably better served as two individual questions. A comparative review splits the focus. Is it likely that either snippet is perfectly clean? No. So both snippets can be served with some clean up. The answer to the question of "which is the cleanest" should be fairly obvious... the cleanest one is probably the one that got less scrutiny when you posted it as an individual question. And the upside is, both snippets received some clean up!

If you're looking for a broad, all-encompassing review of a piece of code, post it by itself. If you're looking for very specific comparison about specific aspects of two concrete alternative implementations, then maybe a comparitive-review is right for you.

---

So, what would make for a good comparative review?

Java method - levels of abstraction

I think this question serves as an example of what could make for a good comparative-review question.

It meets the first criteria described in this post. Both implementations are written in the same language (and do the same thing). Any language expert looking at this question could equally judge both implementations.

It meets the second criteria described in this post. The question is not about which is faster, which is more memory efficient, etc. If this were a question, the asker has already tested and determined the difference is insignificant.

It meets the third criteria described in this post. The criteria by which reviewers should judge the compared snippets is quite clear. In this specific example, we're asked to compare the level of abstraction of the two snippets. Is A not-abstracted enough? Or is B too abstracted?

Importantly, this review might not make very much sense if each snippet were separated into different questions.

The user could've posted snippet A as "Is this abstract enough? How can I make it more abstract?" and snippet B as "Is this too abstract? How can I make it less abstract?" but then the answers will focus as much or more on how to make the snippets more/less abstract and less on how to decide on the right balance. It would be essentially impossible for the user to receive the same answer on both questions if he posted the snippets separately! In fact, it seems likely that question A is the answer to question B and question B is the answer to question A!

With any sort of a performance comparison, is anyone actually trying to strike a balance? No. What the asker actually wants is the most optimally performing snippet of code. If the user presents snippet A and snippet B, the answer to the question might be to implement snippet C, which the asker never even considered. So, rather than an overly complicated question which first asks which is more optimal between two less-than-C-optimal versions of the code, in which any answer must at a minimum feel obligated to spend time discussing the performance difference between A and B, wouldn't it be better if we just expected askers to come knowing what of all their attempts has the most optimal performance, and instead ask how to make it more optimal?

And with any sort of "which is better?" question, isn't this too broad? Isn't this not seeking a review? Without appropriate, specific guidelines, isn't this primarily opinion based (even moreso than an average Code Review question)? Isn't this potentially just seeking an explanation of the snippets?

---

This answer defines a very narrow spectrum off on-topic comparative-review questions. But I believe it defines a spectrum that will lead toward very high quality comparative-review questions, and should encourage others (who might have otherwise posted a comparative-review question) to do a little more research and ask better, more direct, more interesting non-comparative-review questions.

Answer 4

I'm going to buck the conventional wisdom here and suggest that these questions might not be on topic.

Code Reviews happen just before code gets merged in (think GitHub's Pull Request). The code is written. It is complete and has been tested. I certainly wouldn't have two versions of the same method/class in my local repository when submitting a pull request to a project. At that time, I would have had to have implemented either one or the other. I had to have picked one. The other may have been written, but I decided to go with the other, so it only exists in the ether of the log history. Doesn't this mean one of the two snippets is hypothetical and not really code in my project? Doesn't this mean that we're really helping someone make a whiteboard design decision instead of reviewing their code?

These questions also have the potential to be primarily opinion based. I think we've done a good job as a community of writing answers based on objective factors, but it still opens that door as well.

I'm not saying that we need to go close all of the comparative-review questions right now, but we should come back to think and talk about this. I, for one, am not convinced these questions are really on topic. If they are, then they are not very good questions in my opinion.