Note: By time of posting of this meta question, the question in question wasn't closed yet. It is now.
Should we evaluate it on a case by case basis?
To the best of my knowledge, that's how it currently goes. Whether we should, well, it's a messy business which I'll attempt to explain later on. So I'm not sure it's avoidable. We simply can't fit every situation in a flowchart, people will find a way to create a new situation and for some reason it might be beneficial to stray from the normal path.
There are 2 problems here. 1 of them is answer invalidation, another is answering off-topic questions. We can all agree off-topic questions should NOT be answered [2], in part due to the mess I'm going to describe.
I've been involved with quite a lot of rollbacks due to answer invalidation. Answer invalidation is quite a problem on Code Review. There are some relevant meta's I'll link [1], [2], but the short story is not rolling back usually allows a mess to be created.
Consider the following scenario:
- User1 posts question
- User2 posts answer
- User3 posts answer
- User1 changes question. Either because the original was wrong, it was lacking important information, it was example code, filled with bugs or whatever reason you can think of.
- All of a sudden the answer of User2 no longer makes sense. We got answer invalidation. The answer by User3 still applies partially, and since User3 immediately notices the edit by User1, attempts to fix his answer by changing it to the new situation. Seeing new opportunities, User3 fleshes out his answer a little more to be even more clear and more complete.
- In the meantime, due to the amount of answers and edits taking place, the question hits the Hot Network Questions list and attracts a lot of viewers with more possible answers incoming.
What should we do?
A rollback will fix the immediate issue, but now the answer by User3 does no longer make sense. A rollback on that answer would fix that too, but now work from 2 authors goes to waste. Regardless of the action taken, it will still be a mess.
And no, this is not an entirely hypothetical scenario. We've seen this kind of stuff happen.
Answer invalidation is bad. Off-topic questions should be closed. But sometimes, we're not fast enough and a mess is created.
What happened with that question was, IMO, the best thing that could've happened with it. Disclaimer: I was part of it and could've responded even better. I failed to vote-to-close. It got rolled back, the answerer got notified not to answer off-topic questions and the question got rightfully closed. It's up to the author of the question to post a new question now, within CR's scope, if he so pleases.
What specifically happens with such questions, has to be considered on a per-question basis. However, we can't change the rules just to try and avoid a mess. If an answer is posted to a question that shouldn't have been posted, that's unfortunate. We will close the question. The answer doesn't necessarily have to be removed (although the question still risks being deleted if it's awful enough, answered or not).
Or should we allow answer invalidation if it brings an otherwise off-topic question on-topic? It's not my intention to cast aspersions on anyone (I've answered broken code before) but the answer was premature.
Absolutely not. Close the question, request a follow-up. The answer was premature indeed and shouldn't have been posted. Policy is to downvote those answers, but this is not often enforced in my experience.