Should votes to close a question as containing broken code be paired with a comment?

First off; I'm not looking to enforce this via systems; just as a community guideline.

As an example to showcase my issue, here's a review that may pop up in the review queue for Close Votes. It's close-voted as "broken code". Take a moment to consider why or how it is broken. I'm not going to give you any context here; there wasn't any relevant context regarding brokenness on the real question.

import java.util.*;
public class Stack2{
private int[] stack;
private int size;

public Stack2(){
stack = new int[10];
size = 0;
}

public Stack2(int height){
if(height <= 0){
throw new EmptyStackException();
}
stack = new int[height];
size = 0;
}

if (size == stack.length){
throw new StackOverflowException();
}
stack[size] = value;
size++;
}

public int pop(){
if(size == 0){
throw new EmptyStackException();
}
size--;
return stack[size + 1];
}

public int peek(){
if(size == 0){
throw new EmptyStackException();
}
return stack[size - 1];
}

public boolean isEmpty(){
if(size == 0){
return true;
}
return false;
}

public int getSize(){
return this.size;
}

class StackOverflowException extends RuntimeException{
public StackOverflowException(){
super("Nothing can be added to the stack. The stack was full and has overflowed");
}

public StackOverflowException(String message){
super(message);
}
}
}


It's a question, if that helps.

Alright. Found the bug? How long did it take?

Personally, I didn't find the bug. I took about 20 seconds to look at this code, look it through a bit. Didn't see anything obviously wrong.

The bug is, apparently, quite obvious if you test the code:

This code does not behave as expected (add(3);pop() returns 0).

(Now that we've seen this question in detail, here's the link for the question in review and here's the question itself)

The close voter has, luckily, added a comment!

It has been demonstrated in an answer, that this code does not behave as expected (add(3);pop() returns 0). As such I'm voting to close this question as off-topic because the code is not working as intended. For more information, please see the help center. Thanks - Vogel612

As a result, after I didn't see the bug on my first pass, I read the comments, including the comment containing the reason to vote to close. I scrolled back up and verified the bug, and could then vote to close with certainty. A stack that doesn't give the right items back is just broken.

However...

There are cases where someone votes to close a question because it contains broken code... and doesn't add a reason. And I'm left baffled. The code looks fine. In some of those cases I'll skip; in others I'll vote to leave open because there was no reason provided.

My question; should we make it a community guideline that if you vote to close a question based off the fact that the code is broken, you add a comment explaining why you do so?

This would allow for two things: One, to weigh by your own merits whether the bug is a major flaw or an edge case ("Your calculator fails for MAX_INT!") and two, to more easily SEE what's wrong with the code. Helps streamline reviews.

A counter argument could be that people should try compiling the code, but again, what if it compiles just fine (the code in this question does!)? Should I test it by hand too? That'd take too much time. (Plus, what about code snippets that need a framework to test?) I can test parts of the code in my head, if you'd just highlight the area for me.

• Do you wan to do this for questions where the author admits it's broken as well? – Mast Jul 27 '16 at 12:11
• @Mast I don't know. I think it would help, but I know that some of those are just slam dunks where you say "look, the title contains 'not working', you're talking about a stacktrace... it's broken." For now, I'm arguing for the thing I want most; that there are explanatory comments on VTC broken code when the main reason is to be found in the actual code. – Pimgd Jul 27 '16 at 12:14
• I think this might be a good practice. It gets complicated. I just edited a question where the author had code that worked for what it was originally designed for then said it didn't work for an extended case, then had additional code that to improve it, but was throwing errors. – pacmaninbw Jul 27 '16 at 12:18
• Umm... isn't broken code supposed to be fixed and not closed? – EKons Aug 10 '16 at 8:48
• @ΈρικΚωνσταντόπουλος By answerers? – Pimgd Aug 10 '16 at 8:57
• @Pimgd Yeah, that's the reason for CR AFAIK... If it isn't, then go like this: SO -> CR... – EKons Aug 10 '16 at 8:58
• @ΈρικΚωνσταντόπουλος It most definitely isn't, because... what the hell is SO for if you can just say "heeey my code is broken please fix" on Code Review? – Pimgd Aug 10 '16 at 9:17
• @Pimgd Yeah, I knew.... anyways, the question should then be deleted until the code is fixed... But then there is the edge-case where someone actually wants such "broken" behavior. – EKons Aug 10 '16 at 9:19
• "Forward reference": meta.codereview.stackexchange.com/questions/6975/… – Vogel612 Aug 15 '16 at 14:27

Yes

I am a comment lover. Comments helps a lot. Especially when it's not crystal clear for everybody that the code is not working as intended.

Sometimes it's even good if two or more users independently considers the code not working as intended and adds a comment about that. (Just to avoid five close votes because "one person says so")

The benefits of a comment together with a close vote is that you can more easily determine if the comment is correct, otherwise you would have to guess and spend a long time looking through the code and think for yourself "Why did someone vote to close this?"

Note: I am only answering the question of "Should you add a comment if you vote to close", I am not trying to answer "Should you vote to close"

• I upvote this because this answer displays pretty much what I was thinking. – Pimgd Jul 27 '16 at 15:47
• I accepted this answer because it best displays the general consensus displayed in chat - yes, commenting is optional, hence "yes it should", not "yes it must". – Pimgd Jul 28 '16 at 15:00

As the moderator involved, I'm obligated to respond.

This specific question

I have to admit, it was a complex decision process for me, with many factors to consider.

1. Code Review is a place where programmers post their code for feedback…
2. … subject to a condition that the code must work as intended, to the best of the author's knowledge.

If the question specifically asks for help to fix a bug, then it's instantly off-topic. However, we have traditionally taken the position that questions with obviously broken code should be closed, too, so that users cannot skirt the rule by claiming ignorance.

But what is "obvious"? If the code cannot possibly compile, I'd consider it obviously broken. If even the simplest usage of the code would produce incorrect results, then there would also be a strong case for considering it to be "obviously broken".

For the Stack2 in question, that was the criterion I used when hammering the question closed at 2016-07-26 18:00:40Z. It seemed likely that the author hadn't bothered to test the code at all before posting. I left the following comment:

Have you tested this code for correctness? Why is there a StackOverflowException class that you never use? Would you like to post your EmptyStackException class? — 200_success♦ 2016-07-26 18:01:51Z

(It turned out that my remarks about the exceptions were unfounded.)

3. Since the author replied to my comment but didn't take the hint to test the code for correctness, I took that as an indication that she was unaware of the brokenness of the code. I therefore reopened the question at 19:16:28Z and wrote an answer at 19:22:48Z that mentioned the serious bug.2

4. Could this matter have been better handled in some other way? By leaving a comment, perhaps?

I've held the position that reviewing should be done in answers rather than in comments. A bug report constitutes an answer. Users who invest time to look at code and find bugs should have the opportunity to earn reputation from it — that's the basic premise of the site.

For a moderator who wields a hammer to close the question, is there more of an obligation to explain the closure with a comment? Perhaps so, and I often do. In this case, I felt that I had fulfilled my duty when asking "Have you tested this code for correctness?" and being quick to reopen.

So, that was my reasoning for closing, reopening, and answering. Five community members have now voted to close the question after all. Should it remain closed? Maybe, if they all agree that the brokenness was "obvious" enough that the question should not have been eligible for review.

General guidelines for commenting

The Meta question here is whether there should be a community guideline for commenting when closing as broken. As you can see, it's not a simple question. Factors include:

• Preserving the question-and-answer nature of the site vs. collaborating to improve the question
• Fairly conferring reputation for spotting bugs vs. altruistic free advice
• Desire to help the author vs. the moral hazard for fixing broken code
• Allowing some degree of reputation gamification (which is what makes the site successful) vs. transparency in justifying closures

A lot depends on your political leanings, and, frankly, each user's mood at the time. I don't think that there is a guideline that could satisfy everyone. If there were such a guideline, I don't think I could help enforce it, let alone abide by it consistently myself.

1 See also: my opinions on Should minor broken code be reviewed? and How active should moderators be in opening/closing questions?

2 This action, too, carried a moral dilemma. As a moderator with the ability to close and reopen questions at will, I would have a first-mover advantage when answering a freshly reopened question. I often self-impose a waiting period between reopening and answering, but it was bedtime for me, and I didn't want to leave this unfinished business overnight.

• But often when code is broken, reviews turn into a disappointment as half of the review is wasted due to the code actually having to function differently. That's what bothers me about this; keeping questions with obvious or near-obvious bugs in them opened seems to result in a waste of time of practically everyone involved... – Pimgd Jul 27 '16 at 21:42
• @Pimgd But that is the nature of asking for reviews from strangers who are rewarded by nothing more than fake Internet points. The more effort you put into crafting the question, the more rewarding your experience should be. – 200_success Jul 27 '16 at 21:45
• I'm saying it's a waste of time for the reviewers too, finding out half your review is useless because actually the code doesn't work at all for any input – Pimgd Jul 27 '16 at 21:46
• @Pimgd The waste of time was already going to happen as soon as the faulty code was posted. As a reviewer, why should I spend time to spot the bug, close it (for no reward), explain what needs to be fixed (for no reward), wait for the question to be reopened before posting my observations, and hope that in the meantime nobody else beats me to it? You post your code, and that's the code that gets reviewed — that's the basic premise of the site. And my answer wasn't useless: spotting an off-by-one error was absolutely of value to the author. – 200_success Jul 27 '16 at 21:52
• Because the expected utility (or reputation) from closing a question after you spot a bug in it within the first minute and then moving on to another question beats the expected utility from answering anyway, permanently marking the question as a bad one (no editing the code after getting answers!), limiting the upvotes you could get? – Pimgd Jul 27 '16 at 21:54