Do we want to enable the review auditing feature that discourages robo-reviewing?

Question

For all of you who don't know what review audits are, have a read of this stackexchange meta question: what-are-review-tests-audits-and-how-do-they-work

This question is related to these code review meta questions

• Can we suggest a review of a reviewer, or an evaluation of a person using the review queue heavily?

• do-we-want-site-specific-editing-guidance

• and for sure some more ;-)

So the overall question is: do we want to enable the review auditing feature for Code Review ?

Answer 1

No

Audits are a premature solution to a problem that's not particularly bad, and will also be fixed by other solutions "soon".

Currently the reputation needed to participate in the review queues is 500 rep. In full sites the reputation needed is 3000

Fixing the problem by introducing review audits is a backward process. The problems in the review queues are mostly related to people who are not experienced enough with the site doing things that "real" sites only allow after more experience is gained.

Edit: Let me expand this "NO" answer some more. Here are reasons why we don't want audits:

1. The current reviewers are not doing too badly, all things considered. The problem questions are few and far between. The moderators themselves have had more issues with question closes than the site members.
2. They don't work very well. Everyone who's been through StackOverflow's queues can see them from miles away, and they are ignored. They don't discourage robo-reviewing, they do the opposite. Roboreviewers still exist on StackOverflow.
3. They don't work very well. Audits slow down people who actually know what they are doing.
4. They are not needed: How many people have been manually banned from the review queues? In my time as a mod, I believe there were 2, maybe 3 people banned... and for short periods. How many people are currently review-banned?
5. The problem is not that we have robo-reviewers, the problem is that we have people who don't know what's off-topic, or not. The solution is to have reviewers that know the site better, not those who can identify an audit question better.
6. The solution already exists - the whole point of reputation on the site is to allow more site-experience people access to more site-responsible activities. The Systems are reasonably well tuned for graduated sites. 3000 reputation takes a while to earn. 500 is possible in just a few days. For a beta site, the 500 rep is reasonable, the 3000 rep is not.
7. Audit questions creates additional burden for site administrators - audit questions have to be selected, and the meta-site spike in activity as they get challenged, questioned, etc. will be a headache.

So, the proposed audit system does not significantly improve the problem, it irritates the people who are not a problem, and it is more work, all for a problem which is solved by having more experienced reviewers.

@janos - your question/comment: I thought the question is about adding audits. Yes, fixing the privilege levels is the single most important thing we need right now. But I don't see why we are mixing that topic into this discussion of audits. I don't see why you don't see it. To me it is "obvious".

Answer 2

Yes

Audits will make reviewers more careful. Even after the privilege levels are corrected (as they should be on a graduated site), the audits will always have the positive effect of making reviewers more careful. That's a very good thing, isn't it?

In response to the points in @rolfl's answer:

• The current reviewers are not doing too badly, all things considered. -> the audits will help them do even better.

• The queues where we have bigger problems are not queues where there are audits anyway -> so they may help on only one queue, not all of them. A small net positive is still a net positive.

• They don't work very well. Everyone who's been through StackOverflow's queues can see them from miles away, and they are ignored. -> They work on me. So it's not "everyone". They don't annoy me, but I'm not going to say they annoy "no one"

• Roboreviewers still exist on StackOverflow. -> Sadly they probably always will. But how many more would there be without the audits? We cannot really know, only speculate.

• They don't work very well. Audits slow down people who actually know what they are doing. -> What's the rush? This doesn't seem to be a big problem at all.

• They are not needed: How many people have been manually banned from the review queues? -> Our user base is growing rapidly. Problems will grow together with it.

• The problem is not that we have robo-reviewers, the problem is that we have people who don't know what's off-topic, or not. The solution is to have reviewers that know the site better, not those who can identify an audit question better. -> The audits will help educate users.

• The solution already exists - the whole point of reputation on the site is to allow more site-experience people access to more site-responsible activities. The Systems are reasonably well tuned for graduated sites. 3000 reputation takes a while to earn. 500 is possible in just a few days. For a beta site, the 500 rep is reasonable, the 3000 rep is not. -> Guess what, long after having 20k, I still didn't know the rules well. I was too busy earning rep, and not paying much attention to participating in moderation activities. Rep level doesn't always imply a good understanding of the site, and in my case it didn't. The solution is not perfect, it could use a little extra help.

• Audit questions creates additional burden for site administrators - audit questions have to be selected, and the meta-site spike in activity as they get challenged, questioned, etc. will be a headache. -> Thanks for caring. I think we can handle it. But maybe it will be a pita and you'll turn out to be right.

Granted, the audits will not be a groundbreaking "miracle drug". But I think they will be easily a net positive. All the potential negatives seem negligible or highly speculative.

Answer 3

I agree audits sound like a good idea. However, if adding comments on on-topic questions will make the audits fail, I feel the drawback is bigger than the gain and we'll have to look for a better solution to counter robo-reviewers.

Comments are used extensively on Code Review. If the audit tool doesn't understand the value of comments, that tool is not for us.

It isn't always clear whether a question is on-topic. If the OP isn't sure about whether his code works or not or when the wording of the question doesn't specify whether OP tested his own code and how, I tend to leave a comment and check back later. I'm not saying that's how it should be done, but I'm probably not the only one. Audits don't cover grey areas while I've seen a lot of grey posts coming by in the queue. If we want audits, we better have a plan how to do it right.

Answer 4

Yes

I think having review audits will sharpen our skills for doing good reviews. It will lead to having a closer look at the question/answer before we take action.

IMO this will improve the site's health and quality. It will prevent that reviewers are taking action to fast.

But

we need to take into account that continuing the style to add a comment (on an on topic question of a new user) like

Welcome to Code Review. I hope you get some good answers.

will trigger a failure on an audit for a good on topic question.