Confusing phrasing of the "off-topic" close reason

Question

Before asking my last question, I've double checked what counts as off-topic and on-topic in this SE. According to my findings:

We require that the code be working correctly, to the best of the author's knowledge, before proceeding with a review.

I find this poorly phrased compared to what is actually happening. From Gareth Rees comment I take it that any question that is bug related is immediately off-topic.

This is not what is reflected by this phrase. In fact this meta post seems to suggest:

[Purpose of a code review:] Are there edge cases that may cause the code to fail

I would assume this to be on-topic on a code review platform? I am confused because this doesn't seem to be the case.

As such I want to suggest to phrase the off-topic reason more explicitly to reflect current policies. My ideas are:

We require that the code be working correctly before proceeding with a review

or

We require that the code be working correctly in all cases and to the best of the author's knowledge before proceeding with a review

But there are probably better ways. My goal is to make it more clear that anything that is about unexpected behavior is off-topic; including corner cases.

Answer 1

Unfortunately I agree with the close reasons. To the best of your knowledge, your code fails the google test cases, therefore it does not work. Just because you don't know what the problem is, does not mean you don't know there is a problem. Further, just because you don't know what the failing test case is it does not mean that the code works.

On code review we have a specific exception for failing test cases in "competitive" challenges for time-limit-exceeded - questions that get the right result, but take too long - performance problems. You can't even tell us if that's the failing test case.

In other words, you've brought us a question with code that works in your test cases, but your test cases are incomplete, and there are other test cases but you can't tell us what they are, which ones are failing, and what the failing inputs are. You are specifically asking us to determine why those tests (that you can't provide) fail.

Now, if you could give us the failing tests cases, then the question would still be off-topic here, because you should be taking it to StackOverflow instead - with a simplified minimal code example illustrating the problem.

Code Review is for reviewing and improving working code. Sometimes that means discovering/mentioning issues where edge-cases may cause problems (for example, unexpected input may cause SQLInjection problems, or integer overflows, etc.). Those are problems the question asker was unaware of when they asked the question.

Put differently, Code Review is for code that in the real world is "ready for release in to production" - code that you are confident will work, but just want an objective opinion on any improvements that can be made.

Edit: the close-reason contains a link to the "Broken code" answer on the question: My question was closed as being off-topic. What are my options?. I have edited that answer here to include failing test cases as a close reason. This does not help with confusion before asking a question, but it may help with confusion after a question is closed.

Answer 2

The phrasing in the close reason is a long-fought-over compromise.

The Scylla and Charybdis that we are trying to avoid are:

1. We don't want to turn into a debugging service where people post broken code and ask us to fix it. That's a thankless job for Stack Overflow.

2. But we also don't want to have to reject essentially working programs that fail in some corner case.

The close reason is phrased so as to tersely steer a course between these rocks. If you have a suggestion for better phrasing, go for it.

In the particular case in hand, we know that debugging code for programming challenges can be frustrating because the challenge site does not tell you which test case failed, or why. But if you can't figure it out for yourself, then you need Stack Overflow, not Code Review.

Answer 3

Phrasing things is difficult. I want to make it clearer what is currently on-topic and off-topic.

• I have a bug in my code, please help me fix it. (Off-topic)
• Is there any bug in my code? (On-topic)
• Is there any edge-case that I haven't thought about? (On-topic)
• My tests work fine, but the tests that the online judge has fail because of wrong result (Off-topic)

In your question you wrote:

Since my local tests all pass, the remote tests fail and Google doesn't seem to reveal them, can somebody please look at my code and tell me where I am wrong?

Because of Google being the online judge and "the remote tests fail" then this means that the code is not working correctly to the best of your knowledge. You know that your code is not producing the correct results, you just don't know why.

Answer 4

I want to present an entirely different solution, which just occurred to me, and let the community vote on it.

The idea is that we require questions to supply working code anyway. We could get a lot more value out of this by taking a test driven approach to this and slightly extend requirements to include a "working/broken" "test case" that fits the aim of this SE:

Code Review is a community where programmers peer-review your working code to address issues such as security, maintainability, performance, and scalability.

If I have an issue with performance or scalability, I can probably create an example to demonstrate the issue. I did have a performance example of this recently: here I ended up providing such a test case to create a framework for comparability. There was a bug in my initial code; however, it was unrelated to the test case so I guess that's what makes it okay?

I never used this SE for security questions, but I can imagine that something like "is it possible to exploit X to achieve Y" can be turned into a test bed that can be provided by an OP and would make it easier to answer concisely.

Maintainability is a rather subjective goal imho and (for me) it is always challenging to figure out if it is opinion based or not. I generally refrain from asking these questions, maybe someone else can help here.

An advantage of such an approach would be that it would clearly say that questions like this are okay, but this isn't (in lack of being able to provide a test of what is broken).

Further, we can clearly say things like: "reviewing to pass this kind of test is not the aim of this SE; the question doesn't test X on performance but on a random corner case. Please post it on stackoverflow"

Another advantage is that the answers can focus on improving said performance metric / test and, even if a small corner case bug exists, that can be ignored without wronging the question or answer. (Providing invariance to some smaller or bigger bugs which seems to be wanted here)

---

I have edited that answer here to include failing test cases as a close reason.

As per @rolfl 's edit: This is kind of contradictory to my idea :D Maybe it can still start a discussion and result in something useful.