Should we allow answer invalidation for otherwise off-topic questions? [duplicate]

Question

We recently had a question with un-compilable code. The question was answered and the answerer pointed out the broken code. The user has since edited the question to fix the code. It was rolled-back.

But my question is: Should we?

Or should we allow answer invalidation if it brings an otherwise off-topic question on-topic? It's not my intention to cast aspersions on anyone (I've answered broken code before) but the answer was premature.

While it is not the case in this scenario, my immediate concern with making this policy would be that making the code work might require drastic changes to the code. This could get very messy very fast.

Should we evaluate it on a case by case basis?

Answer 1

Let me quote the answer to which @Mast posted a link in his answer:

Answers to off-topic questions are already invalid

One way to look at this is to say that if a question is invalid, it should be closed and not answered. Answering an invalid question sends exactly the wrong message and encourages unwanted behavior.

Following that line of reasoning, fixing the question to make it on topic doesn't invalidate any answer because any answers were already invalid.

This accepted answer has 18 up-votes and one down-vote hence we could say that we have achieved a consensus.

Althought the question hadn't been closed at the point of answering it, the answerer pointed out the problems and with doing so he/she knew that the code in question won't work in its current state, hence the answer had been invalid and editing the code was fair game without needing roll-backs.

Answer 2

Note: By time of posting of this meta question, the question in question wasn't closed yet. It is now.

Should we evaluate it on a case by case basis?

To the best of my knowledge, that's how it currently goes. Whether we should, well, it's a messy business which I'll attempt to explain later on. So I'm not sure it's avoidable. We simply can't fit every situation in a flowchart, people will find a way to create a new situation and for some reason it might be beneficial to stray from the normal path.

There are 2 problems here. 1 of them is answer invalidation, another is answering off-topic questions. We can all agree off-topic questions should NOT be answered [2], in part due to the mess I'm going to describe.

---

I've been involved with quite a lot of rollbacks due to answer invalidation. Answer invalidation is quite a problem on Code Review. There are some relevant meta's I'll link [1], [2], but the short story is not rolling back usually allows a mess to be created.

Consider the following scenario:

1. User1 posts question
2. User2 posts answer
3. User3 posts answer
4. User1 changes question. Either because the original was wrong, it was lacking important information, it was example code, filled with bugs or whatever reason you can think of.
5. All of a sudden the answer of User2 no longer makes sense. We got answer invalidation. The answer by User3 still applies partially, and since User3 immediately notices the edit by User1, attempts to fix his answer by changing it to the new situation. Seeing new opportunities, User3 fleshes out his answer a little more to be even more clear and more complete.
6. In the meantime, due to the amount of answers and edits taking place, the question hits the Hot Network Questions list and attracts a lot of viewers with more possible answers incoming.

What should we do?

A rollback will fix the immediate issue, but now the answer by User3 does no longer make sense. A rollback on that answer would fix that too, but now work from 2 authors goes to waste. Regardless of the action taken, it will still be a mess.

And no, this is not an entirely hypothetical scenario. We've seen this kind of stuff happen.

Answer invalidation is bad. Off-topic questions should be closed. But sometimes, we're not fast enough and a mess is created.

What happened with that question was, IMO, the best thing that could've happened with it. Disclaimer: I was part of it and could've responded even better. I failed to vote-to-close. It got rolled back, the answerer got notified not to answer off-topic questions and the question got rightfully closed. It's up to the author of the question to post a new question now, within CR's scope, if he so pleases.

What specifically happens with such questions, has to be considered on a per-question basis. However, we can't change the rules just to try and avoid a mess. If an answer is posted to a question that shouldn't have been posted, that's unfortunate. We will close the question. The answer doesn't necessarily have to be removed (although the question still risks being deleted if it's awful enough, answered or not).

Or should we allow answer invalidation if it brings an otherwise off-topic question on-topic? It's not my intention to cast aspersions on anyone (I've answered broken code before) but the answer was premature.

Absolutely not. Close the question, request a follow-up. The answer was premature indeed and shouldn't have been posted. Policy is to downvote those answers, but this is not often enforced in my experience.