11
\$\begingroup\$

For all of you who don't know what review audits are, have a read of this stackexchange meta question: what-are-review-tests-audits-and-how-do-they-work

This question is related to these code review meta questions

So the overall question is: do we want to enable the review auditing feature for Code Review ?

\$\endgroup\$
  • \$\begingroup\$ If I remember correctly, this is something that is done on graduated sites. I am not sure if, or when, this will be a part of our "Design Independent Graduation" \$\endgroup\$ – Simon Forsberg Oct 26 '15 at 10:49
  • 1
    \$\begingroup\$ Please see: meta.stackexchange.com/a/199195/232788 @SimonForsberg \$\endgroup\$ – Heslacher Oct 26 '15 at 11:01
  • 1
    \$\begingroup\$ Note that moderators occasionally check that reviewers are doing their job responsibly. \$\endgroup\$ – 200_success Oct 26 '15 at 17:31
  • \$\begingroup\$ As just a side note: I noticed some poor-quality reviews from @t3chb0t. If you see his activity page there are some low-quality reviews, especially the most recent one (at the time of writing), which was a code edit not by OP. This just goes to show that maybe audits are required... \$\endgroup\$ – TheCoffeeCup Nov 1 '15 at 19:38
11
\$\begingroup\$

No

Audits are a premature solution to a problem that's not particularly bad, and will also be fixed by other solutions "soon".

Currently the reputation needed to participate in the review queues is 500 rep. In full sites the reputation needed is 3000

Fixing the problem by introducing review audits is a backward process. The problems in the review queues are mostly related to people who are not experienced enough with the site doing things that "real" sites only allow after more experience is gained.

Edit: Let me expand this "NO" answer some more. Here are reasons why we don't want audits:

  1. The current reviewers are not doing too badly, all things considered. The problem questions are few and far between. The moderators themselves have had more issues with question closes than the site members.
  2. They don't work very well. Everyone who's been through StackOverflow's queues can see them from miles away, and they are ignored. They don't discourage robo-reviewing, they do the opposite. Roboreviewers still exist on StackOverflow.
  3. They don't work very well. Audits slow down people who actually know what they are doing.
  4. They are not needed: How many people have been manually banned from the review queues? In my time as a mod, I believe there were 2, maybe 3 people banned... and for short periods. How many people are currently review-banned?
  5. The problem is not that we have robo-reviewers, the problem is that we have people who don't know what's off-topic, or not. The solution is to have reviewers that know the site better, not those who can identify an audit question better.
  6. The solution already exists - the whole point of reputation on the site is to allow more site-experience people access to more site-responsible activities. The Systems are reasonably well tuned for graduated sites. 3000 reputation takes a while to earn. 500 is possible in just a few days. For a beta site, the 500 rep is reasonable, the 3000 rep is not.
  7. Audit questions creates additional burden for site administrators - audit questions have to be selected, and the meta-site spike in activity as they get challenged, questioned, etc. will be a headache.

So, the proposed audit system does not significantly improve the problem, it irritates the people who are not a problem, and it is more work, all for a problem which is solved by having more experienced reviewers.

@janos - your question/comment: I thought the question is about adding audits. Yes, fixing the privilege levels is the single most important thing we need right now. But I don't see why we are mixing that topic into this discussion of audits. I don't see why you don't see it. To me it is "obvious".

\$\endgroup\$
  • 2
    \$\begingroup\$ This. Cure the disease, not the symptom. \$\endgroup\$ – RubberDuck Oct 26 '15 at 15:24
  • \$\begingroup\$ 3000 rep is what's needed for Close and Reopen vote queues, but first and last posts are accessible at 500 rep and Low Quality/Suggested Edits are reviewable at 1500 rep. \$\endgroup\$ – SuperBiasedMan Oct 27 '15 at 11:31
  • 2
    \$\begingroup\$ @SuperBiasedMan - correct (well, almost, suggested edits are at 2000, not 1500), but those are more "reasonable" than the current state at 350 and 1000. \$\endgroup\$ – rolfl Oct 27 '15 at 12:23
  • \$\begingroup\$ Indeed, just trying to correct on numbers (and getting it half right). \$\endgroup\$ – SuperBiasedMan Oct 27 '15 at 12:26
4
\$\begingroup\$

Yes

Audits will make reviewers more careful. Even after the privilege levels are corrected (as they should be on a graduated site), the audits will always have the positive effect of making reviewers more careful. That's a very good thing, isn't it?

In response to the points in @rolfl's answer:

  • The current reviewers are not doing too badly, all things considered. -> the audits will help them do even better.

  • The queues where we have bigger problems are not queues where there are audits anyway -> so they may help on only one queue, not all of them. A small net positive is still a net positive.

  • They don't work very well. Everyone who's been through StackOverflow's queues can see them from miles away, and they are ignored. -> They work on me. So it's not "everyone". They don't annoy me, but I'm not going to say they annoy "no one"

  • Roboreviewers still exist on StackOverflow. -> Sadly they probably always will. But how many more would there be without the audits? We cannot really know, only speculate.

  • They don't work very well. Audits slow down people who actually know what they are doing. -> What's the rush? This doesn't seem to be a big problem at all.

  • They are not needed: How many people have been manually banned from the review queues? -> Our user base is growing rapidly. Problems will grow together with it.

  • The problem is not that we have robo-reviewers, the problem is that we have people who don't know what's off-topic, or not. The solution is to have reviewers that know the site better, not those who can identify an audit question better. -> The audits will help educate users.

  • The solution already exists - the whole point of reputation on the site is to allow more site-experience people access to more site-responsible activities. The Systems are reasonably well tuned for graduated sites. 3000 reputation takes a while to earn. 500 is possible in just a few days. For a beta site, the 500 rep is reasonable, the 3000 rep is not. -> Guess what, long after having 20k, I still didn't know the rules well. I was too busy earning rep, and not paying much attention to participating in moderation activities. Rep level doesn't always imply a good understanding of the site, and in my case it didn't. The solution is not perfect, it could use a little extra help.

  • Audit questions creates additional burden for site administrators - audit questions have to be selected, and the meta-site spike in activity as they get challenged, questioned, etc. will be a headache. -> Thanks for caring. I think we can handle it. But maybe it will be a pita and you'll turn out to be right.

Granted, the audits will not be a groundbreaking "miracle drug". But I think they will be easily a net positive. All the potential negatives seem negligible or highly speculative.

\$\endgroup\$
  • \$\begingroup\$ If you help in one place and hinder in others, you might not have a net positive. The audit system sucks, there's plenty cases where it hits you with a stick for doing something like wanting to improve an answer that has been upvoted by people. If you slow me down, I might stop reviewing at all. There's not a problem no so no need for a fix now. \$\endgroup\$ – Pimgd Oct 28 '15 at 11:59
2
\$\begingroup\$

I agree audits sound like a good idea. However, if adding comments on on-topic questions will make the audits fail, I feel the drawback is bigger than the gain and we'll have to look for a better solution to counter robo-reviewers.

Comments are used extensively on Code Review. If the audit tool doesn't understand the value of comments, that tool is not for us.

It isn't always clear whether a question is on-topic. If the OP isn't sure about whether his code works or not or when the wording of the question doesn't specify whether OP tested his own code and how, I tend to leave a comment and check back later. I'm not saying that's how it should be done, but I'm probably not the only one. Audits don't cover grey areas while I've seen a lot of grey posts coming by in the queue. If we want audits, we better have a plan how to do it right.

\$\endgroup\$
  • 4
    \$\begingroup\$ I believe that the audit questions should be very clear, so that it is quite obvious if they're off-topic or good posts. We wouldn't want any grey-area questions in the audits. \$\endgroup\$ – Simon Forsberg Oct 26 '15 at 10:54
  • \$\begingroup\$ Regarding failing because you only added a comment: If it's a good question, you would have already passed because you upvoted... \$\endgroup\$ – Deduplicator Oct 27 '15 at 21:21
1
\$\begingroup\$

Yes

I think having review audits will sharpen our skills for doing good reviews. It will lead to having a closer look at the question/answer before we take action.

IMO this will improve the site's health and quality. It will prevent that reviewers are taking action to fast.

But

we need to take into account that continuing the style to add a comment (on an on topic question of a new user) like

Welcome to Code Review. I hope you get some good answers.

will trigger a failure on an audit for a good on topic question.

\$\endgroup\$
  • \$\begingroup\$ Are you sure that a comment like that will result in a failed audit? I am also wondering how much of the audits that are configurable. \$\endgroup\$ – Simon Forsberg Oct 26 '15 at 10:53
  • \$\begingroup\$ I have come to this conclusion by checking stack exchange meta e.g meta.stackexchange.com/questions/188885/… and also by own experience on stack overflow. \$\endgroup\$ – Heslacher Oct 26 '15 at 11:00
  • \$\begingroup\$ Any comment in an audit is treated as if it's a criticism/correction and therefore only seen as appropriate on a 'bad' post. \$\endgroup\$ – SuperBiasedMan Oct 27 '15 at 11:32
  • \$\begingroup\$ @SuperBiasedMan: The solution is simple: First upvote if it's a good post, then comment. I mean, that comment should come with an upvote, or that comment is actually the wrong action. \$\endgroup\$ – Deduplicator Oct 27 '15 at 21:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .