Code that handles n% of test cases but not all

Question

I've started noticing recently questions like this one (66%, not well received) or that one (50%, well received) where the author states something like this:

I am trying to solve this problem on Codility. I just got 66% for my score. My solution fails at the following cases...

In fact it looks like a lot of them come from Codility users, so we are bound to continue seeing those.

While it's clear that the code "works correctly" to a reasonable extent (say if it's 50% or more), it's also clear that it doesn't "work completely correctly", i.e., it's failing some edge cases and whatnot.

If they are asking for help in improving this score to handle more cases, is that OK here or, is it more Stack Overflow territory?

What would be the consensus on how we can best handle this, whether we consider it on-topic, and how to best communicate that to the question's author?

Answer 1

If the code produces wrong answers for some test cases, then the author is aware that the code is not working correctly as intended, and the question is thus prima facie off-topic for Code Review.

There may be room for certain exceptions, though:

• If it's failing due to time limits rather than correctness, then that is an issue of scalability, which is OK. The time-limit-exceeded tag applies.
• If the answer is just slightly off due to rounding inaccuracies, then I think that it's valid to discuss precision issues as part of a code review.
• If it's 97.5% correct, I would let it slide. That's pretty darn close, and I think it qualifies as "unexpected corner cases". I don't believe that 60% correctness is anywhere near acceptable, though.
• If the author includes copious unit tests of their own as a demonstration that the online judge may be wrong, I might be inclined to be lenient.

Both of the example questions that you cited should be closed, by these standards. Our usual closure comments for broken code apply.

Answer 2

The relevant criterion for whether a question is off-topic states:

To the best of my knowledge, does the code work as intended?

However, in the case of this question, getting 50% on correctness means that the code is not working properly to the best of someone else's knowledge, because the test cases where the asker's code failed are apparently not known. And a third-party claim that the code is incorrect does in no way generate knowledge about the code's functionality if the relationship between the claim of incorrectness and the code's functionality is not disclosed. So the code does work as intended to the best of the author's knowledge. Indeed, the author even admitted to being confused about getting such a low score, because he had thought the task would be "pretty straightforward".

In the other question, however, the one with a 66% assessment, the author seems to have been aware in what particular cases the code failed. So clearly, the code does not work as intended to the best of the author's knowledge. This also makes it clear that Codility's verdict of the code is completely irrelevant for whether a question is off-topic or not: The fact that the author knew of situations where the code does not work is enough to make the question off-topic, regardless of what score the code got on Codility.