14
\$\begingroup\$

I've started noticing recently questions like this one (66%, not well received) or that one (50%, well received) where the author states something like this:

I am trying to solve this problem on Codility. I just got 66% for my score. My solution fails at the following cases...

In fact it looks like a lot of them come from Codility users, so we are bound to continue seeing those.

While it's clear that the code "works correctly" to a reasonable extent (say if it's 50% or more), it's also clear that it doesn't "work completely correctly", i.e., it's failing some edge cases and whatnot.

If they are asking for help in improving this score to handle more cases, is that OK here or, is it more Stack Overflow territory?

What would be the consensus on how we can best handle this, whether we consider it on-topic, and how to best communicate that to the question's author?

\$\endgroup\$
3
  • \$\begingroup\$ Another recent example: Jumps of a Pawn — Java interview \$\endgroup\$ Apr 28, 2018 at 5:23
  • \$\begingroup\$ "In fact it looks like a lot of them come from Codility users", Codility, Hackerrank, basically every site with an on-line judge using multiple test-cases. \$\endgroup\$
    – Mast Mod
    Apr 28, 2018 at 8:50
  • \$\begingroup\$ Previously. \$\endgroup\$ Apr 28, 2018 at 16:56

2 Answers 2

28
\$\begingroup\$

If the code produces wrong answers for some test cases, then the author is aware that the code is not working correctly as intended, and the question is thus prima facie off-topic for Code Review.

There may be room for certain exceptions, though:

  • If it's failing due to time limits rather than correctness, then that is an issue of scalability, which is OK. The tag applies.
  • If the answer is just slightly off due to rounding inaccuracies, then I think that it's valid to discuss precision issues as part of a code review.
  • If it's 97.5% correct, I would let it slide. That's pretty darn close, and I think it qualifies as "unexpected corner cases". I don't believe that 60% correctness is anywhere near acceptable, though.
  • If the author includes copious unit tests of their own as a demonstration that the online judge may be wrong, I might be inclined to be lenient.

Both of the example questions that you cited should be closed, by these standards. Our usual closure comments for broken code apply.

\$\endgroup\$
7
  • \$\begingroup\$ Do you think there should be a minimum correctness % to be deemed on-topic? \$\endgroup\$
    – Phrancis
    Apr 28, 2018 at 8:57
  • 6
    \$\begingroup\$ @Phrancis: Let's not make legislation that we'll later regret. \$\endgroup\$ Apr 28, 2018 at 16:57
  • 4
    \$\begingroup\$ @Phrancis I deliberately declined to set a threshold for the correctness percentage, because there is some room for human judgement. A rough guideline might be: could you ship the code with the known bugs? At 60%, certainly not. At 97.5%, it might be justifiable. \$\endgroup\$ Apr 28, 2018 at 18:54
  • 2
    \$\begingroup\$ My opinion is that if the question asker cannot provide the details of the failing test cases, then it is off-topic.... always. It is not OK to say "It passes 97.5% of the test cases!" If the ONLY failing test cases are Time-limit ones, then that is enough detail, and TLE's are allowed. \$\endgroup\$
    – rolfl
    Apr 29, 2018 at 1:42
  • \$\begingroup\$ "If the code produces wrong answers for some test cases, then the author is aware that the code is not working correctly as intended […]" – not if the author does not know the test cases. After all, there might have been a misunderstanding due to which the author did not intend the code to work for the test cases where the code failed. For example, if the input consists of two numbers that represent a range, and the author assumes that the first number would automatically be the lower bound, then the code would fail if there are test cases where the first number is greater than the […] \$\endgroup\$
    – Stingy
    Apr 30, 2018 at 14:41
  • \$\begingroup\$ […] second, but the code still works as intended. Incidentally, this could have been the case with the second question linked to in this question. \$\endgroup\$
    – Stingy
    Apr 30, 2018 at 14:42
  • \$\begingroup\$ Also, by the last of the standards listed in your answer, the second question might not be off-topic because it includes numerous unit tests. \$\endgroup\$
    – Stingy
    Apr 30, 2018 at 14:51
-1
\$\begingroup\$

The relevant criterion for whether a question is off-topic states:

To the best of my knowledge, does the code work as intended?

However, in the case of this question, getting 50% on correctness means that the code is not working properly to the best of someone else's knowledge, because the test cases where the asker's code failed are apparently not known. And a third-party claim that the code is incorrect does in no way generate knowledge about the code's functionality if the relationship between the claim of incorrectness and the code's functionality is not disclosed. So the code does work as intended to the best of the author's knowledge. Indeed, the author even admitted to being confused about getting such a low score, because he had thought the task would be "pretty straightforward".

In the other question, however, the one with a 66% assessment, the author seems to have been aware in what particular cases the code failed. So clearly, the code does not work as intended to the best of the author's knowledge. This also makes it clear that Codility's verdict of the code is completely irrelevant for whether a question is off-topic or not: The fact that the author knew of situations where the code does not work is enough to make the question off-topic, regardless of what score the code got on Codility.

\$\endgroup\$
2
  • \$\begingroup\$ I don't follow your logic. If someone knows a thing, then tells me that thing, I now know it too. Likewise, if someone else knows my code is badly broken (passes just 50% of test cases), then tells me that, then... I know that, now. Yes, it's not particularly useful if the test cases aren't at least hinted with examples, but that's a problem with the website; in any case, the OP still knows their code is broken, and we can't guess how any more than they can. \$\endgroup\$
    – Nic
    May 7, 2018 at 16:00
  • \$\begingroup\$ @NicHartley I've provided some examples in comments to 200_success' answer here (which 200_success mysteriously never responded to). Maybe I'm a bit too focused on the specific question I defended in my answer here, but I seriously think the code from this question is not broken and the 50% correctness stem from a mere misunderstanding, so I think the question was closed inappropriately. Even according to 200_success' own answer here, the question might not be off-topic, but apparently, 200_success is not interested in his/her judgement being questioned. \$\endgroup\$
    – Stingy
    May 7, 2018 at 20:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .