Should votes to close a question as containing broken code be paired with a comment?

Question

First off; I'm not looking to enforce this via systems; just as a community guideline.

As an example to showcase my issue, here's a review that may pop up in the review queue for Close Votes. It's close-voted as "broken code". Take a moment to consider why or how it is broken. I'm not going to give you any context here; there wasn't any relevant context regarding brokenness on the real question.

import java.util.*;
public class Stack2{
    private int[] stack;
    private int size;

    public Stack2(){
        stack = new int[10];
        size = 0;
    }

    public Stack2(int height){
        if(height <= 0){
            throw new EmptyStackException();
        }
        stack = new int[height];
        size = 0;
    }

    public void add(int value){
        if (size == stack.length){
            throw new StackOverflowException();
        }
        stack[size] = value;
        size++;
    }


    public int pop(){
        if(size == 0){
            throw new EmptyStackException();
        }
        size--;
        return stack[size + 1];
    }

    public int peek(){
        if(size == 0){
            throw new EmptyStackException();
        }
        return stack[size - 1];
    }

    public boolean isEmpty(){
        if(size == 0){
            return true;
        }
        return false;
    }

    public int getSize(){
        return this.size;
    }

    class StackOverflowException extends RuntimeException{
        public StackOverflowException(){
            super("Nothing can be added to the stack. The stack was full and has overflowed");
        }

        public StackOverflowException(String message){
            super(message);
        }
    }
}

It's a javadata-structuresstack question, if that helps.

---

Alright. Found the bug? How long did it take?

Personally, I didn't find the bug. I took about 20 seconds to look at this code, look it through a bit. Didn't see anything obviously wrong.

The bug is, apparently, quite obvious if you test the code:

This code does not behave as expected (add(3);pop() returns 0).

(Now that we've seen this question in detail, here's the link for the question in review and here's the question itself)

The close voter has, luckily, added a comment!

It has been demonstrated in an answer, that this code does not behave as expected (add(3);pop() returns 0). As such I'm voting to close this question as off-topic because the code is not working as intended. For more information, please see the help center. Thanks - Vogel612

As a result, after I didn't see the bug on my first pass, I read the comments, including the comment containing the reason to vote to close. I scrolled back up and verified the bug, and could then vote to close with certainty. A stack that doesn't give the right items back is just broken.

However...

There are cases where someone votes to close a question because it contains broken code... and doesn't add a reason. And I'm left baffled. The code looks fine. In some of those cases I'll skip; in others I'll vote to leave open because there was no reason provided.

---

My question; should we make it a community guideline that if you vote to close a question based off the fact that the code is broken, you add a comment explaining why you do so?

This would allow for two things: One, to weigh by your own merits whether the bug is a major flaw or an edge case ("Your calculator fails for MAX_INT!") and two, to more easily SEE what's wrong with the code. Helps streamline reviews.

A counter argument could be that people should try compiling the code, but again, what if it compiles just fine (the code in this question does!)? Should I test it by hand too? That'd take too much time. (Plus, what about code snippets that need a framework to test?) I can test parts of the code in my head, if you'd just highlight the area for me.

---

Discussion about this question also occured in chat

Answer 1

Yes

I am a comment lover. Comments helps a lot. Especially when it's not crystal clear for everybody that the code is not working as intended.

Sometimes it's even good if two or more users independently considers the code not working as intended and adds a comment about that. (Just to avoid five close votes because "one person says so")

The benefits of a comment together with a close vote is that you can more easily determine if the comment is correct, otherwise you would have to guess and spend a long time looking through the code and think for yourself "Why did someone vote to close this?"

Note: I am only answering the question of "Should you add a comment if you vote to close", I am not trying to answer "Should you vote to close"

Answer 2

As the moderator involved, I'm obligated to respond.

This specific question

I have to admit, it was a complex decision process for me, with many factors to consider.

1. Code Review is a place where programmers post their code for feedback…

2. … subject to a condition that the code must work as intended, to the best of the author's knowledge.

   If the question specifically asks for help to fix a bug, then it's instantly off-topic. However, we have traditionally taken the position that questions with obviously broken code should be closed, too, so that users cannot skirt the rule by claiming ignorance.

   But what is "obvious"? If the code cannot possibly compile, I'd consider it obviously broken. If even the simplest usage of the code would produce incorrect results, then there would also be a strong case for considering it to be "obviously broken".

   For the Stack2 in question, that was the criterion I used when hammering the question closed at 2016-07-26 18:00:40Z. It seemed likely that the author hadn't bothered to test the code at all before posting. I left the following comment:

   _{Have you tested this code for correctness? Why is there a StackOverflowException class that you never use? Would you like to post your EmptyStackException class? — 200_success♦ 2016-07-26 18:01:51Z}

   (It turned out that my remarks about the exceptions were unfounded.)

3. Since the author replied to my comment but didn't take the hint to test the code for correctness, I took that as an indication that she was unaware of the brokenness of the code. I therefore reopened the question at 19:16:28Z and wrote an answer at 19:22:48Z that mentioned the serious bug.²

4. Could this matter have been better handled in some other way? By leaving a comment, perhaps?

   I've held the position that reviewing should be done in answers rather than in comments. A bug report constitutes an answer. Users who invest time to look at code and find bugs should have the opportunity to earn reputation from it — that's the basic premise of the site.

   For a moderator who wields a hammer to close the question, is there more of an obligation to explain the closure with a comment? Perhaps so, and I often do. In this case, I felt that I had fulfilled my duty when asking "Have you tested this code for correctness?" and being quick to reopen.

So, that was my reasoning for closing, reopening, and answering. Five community members have now voted to close the question after all. Should it remain closed? Maybe, if they all agree that the brokenness was "obvious" enough that the question should not have been eligible for review.

General guidelines for commenting

The Meta question here is whether there should be a community guideline for commenting when closing as broken. As you can see, it's not a simple question. Factors include:

• Preserving the question-and-answer nature of the site vs. collaborating to improve the question
• Fairly conferring reputation for spotting bugs vs. altruistic free advice
• Desire to help the author vs. the moral hazard for fixing broken code
• Allowing some degree of reputation gamification (which is what makes the site successful) vs. transparency in justifying closures

A lot depends on your political leanings, and, frankly, each user's mood at the time. I don't think that there is a guideline that could satisfy everyone. If there were such a guideline, I don't think I could help enforce it, let alone abide by it consistently myself.

---

¹ See also: my opinions on Should minor broken code be reviewed? and How active should moderators be in opening/closing questions?

² This action, too, carried a moral dilemma. As a moderator with the ability to close and reopen questions at will, I would have a first-mover advantage when answering a freshly reopened question. I often self-impose a waiting period between reopening and answering, but it was bedtime for me, and I didn't want to leave this unfinished business overnight.