# How is Code Review doing right now?

I’m Grace Note, a Community Manager at Stack Exchange. Normally for most sites we run a community site evaluation, as explained in this Meta Stack Overflow post under Public Beta sites. However, these reviews involve comparing our site content against searches on the internet for competing answers - which doesn’t entirely mesh with what we do on this particular site. For this reason we haven’t been running any community site evaluations on Code Review, and instead perform internal reviews on our own periodic basis. We recently completed one such review and so here’s some feedback for y’all.

## Voting and Reputation Issues

There’s a bit of an issue with voting, and by consequence reputation on the site. Code Golf made a comparison themselves and figured that a major problem was in the lack of reputation by virtue of the lack of voting. Code Review is doing a fair amount better in that department (having 3 users at over 10k) and having a decent spread (21 users can vote to close/reopen, and all 40 of the first page can edit anything), but it dwindles quickly after that. 40 looks cool but the number of avid users for the site is almost a thousand, rendering that as less than 4% of the “active userbase” (more on that later) with decent privileges.

Voting is an odd beast here. When we reviewed the site, the front page had a dearth of voting. Even looking now, in the last 100 questions asked and the last 100 answers posted, less than 10 of each have scores past 3 with several negatively rated. Now, there are users who vote - vote a lot in fact, as most of the top users have hundreds of votes under their belts (with decent downvote rates as well). But in spite of this, there’s still a humongous bed of users sitting in the ~100-200 reputation range and not actually in the major privilege levels. So for where-ever all the votes are going, it ain’t accumulating.

## Community Engagement

One positive point is that this site has an extremely engaged Meta. There’s healthy activity in recent days that surpasses a lot of other sites that we check up for graduation review.

For all that engagement, though, the site itself isn’t keeping people. User retention is less than ideal - there’s almost a thousand users listed as “avid” on Area 51, but according to our traffic metrics less than a hundred of them visit on a regular basis, and less than two hundred were even seen in the last month. These aren’t happy numbers. They are in fact unhappy numbers. Very unhappy numbers.

The majority of this is in the lower reputation brackets. The top reputation users consistently visit the site but as we break down below 2000 and 1000 reputation, the retention rate drops exponentially. By 1000 we’re already at half, and again we lose almost 90% by even just the 150 reputation range. This basically appears to show that people who already have an investment here are sticking around, but few new people truly start big journeys.

Jon Ericson, another Community Manager, ran some numbers on the site and discovered that the average time it takes for a question to receive its first answer is actually several hours (averaging somewhere between 8 and 10 hours). This seemed pretty high, but a quick browse of the questions is pretty enlightening - things are huge here. A lot of questions comprise of gigantic blocks of code that have little guidance as to what should be reviewed. This, however, can be remedied a bit.

• Ask people to provide less gigantic blocks of code and instead scope their code to the relevantly smaller portions
• Ask people to provide instructions and/or guidance as to what they’re looking to get out of the review or what they want to have addressed
• Encourage a stronger positive feedback loop such as by editing

Monstrous code blocks are a two-fold to the site’s potential. It scares away new answerers because of the amount of effort needed to get in (which, with voting distribution, isn’t necessarily rewarding either). And without getting answerers, askers don’t get as much out of the site leaving them to vanish as well. Which brings me to the final point...

Tying in to all of the above is one of the most pressing concerns about the site I had in my own review, which is the gigantic pool of unanswered questions. There are, at the time of this writing, 1237 questions without any upvoted answers, a whopping 1010 of which have absolutely no answers at all. That’s over 10% of the site’s questions, sitting untouched and barely considered to be alive. These questions range from the site’s birth in 2011 all the way to modern day 2013. About 25% of these questions result in users never returning after a week, from a rudimentary query.

This isn’t a small number. This is a huge chunk of the site that’s basically just a black hole of despair, swallowing everything without giving anything back. And there’s a simple enough fix to this - clean up the unanswered questions. A large amount of what I see from these is questions that get a fair amount of comments back-and-forth when first asked, but then the entire dialogue is dropped and no one on either side follows up. The questions essentially become abandoned, not unlike the ultimate fate of this site if people don’t keep up.

In conclusion, the main problem points are the distribution of votes and reputation resulting in a largely bottom-heavy reputation spread that cannot support the site, an exceedingly long average time between answers, and severe question abandonment issues resulting in a lot of unanswered questions now and old.

Past that, though, the site’s actually doing rather well. It has a humongous userbase behind it (albeit, again, bottom-heavy), the userbase is engaged on Meta, traffic metrics are superb, and overall the site has a positive feel. When we’re not looking at the stagnant and decayed questions, I see extremely in-depth reviews and helpful advice, as well as both askers and answerers working together. However, between the lack of voting and the poor question/answer return rate, it isn’t ready to see graduation.

I hope y’all find our findings to be helpful, and let us know if you have any further questions. For example if you yourselves know any particular fall points that you’d like advice on, let us know.

If you've made it this far in here, check out the 100 days update that y'all fished up! It's worth a study and comparison!

• Thank you for reviewing us. I agree with many of these points, especially the voting. First, I'd like to ask about the "Q&Aness" of this site. Would you say that our answering style fits this well enough, or not quite? Code reviews can be general, where no questions are asked. The OP just (hopefully) describes the code and lets us get to work. – Jamal Nov 15 '13 at 8:38
• The second thing is about moderators. My understanding is that CR used to have four moderators, then three, and now two. Based on the size of this site, would you say we have enough moderators, or would an additional one (to replace Winston Ewert) be needed? Unfortunately, we do not have enough trusted users here using their powers (mainly deleting non-answers), so I end up having to do a lot of flagging. Other than that, I feel there should always be at least one moderator around in case there's trouble. What's your take on this? – Jamal Nov 15 '13 at 8:40
• Oh, and... should these questions be posted as comments or as answers? – Jamal Nov 15 '13 at 9:02
• @Jamal Either way works for me. Comments are good for one-shot kinds of things while an answer makes it easier for repeated back-and-forth. Regarding the moderator, we're working on invites for a new moderator as we prefer to have at least 3 moderators per site, but we've yet to get a yes yet. So we're still askin'. – Grace Note Nov 15 '13 at 10:29
• Woah I can't believe it took me 8 hours to see this post!! Thanks for posting this extremely useful review on CR! Do I understand correctly that old, abandonned questions should simply be deleted? – Mathieu Guindon Nov 15 '13 at 14:48
• When you say clean up the unanswered questions what do you exactly mean? Delete or answer them? Deleting them isn't the correct way I am assuming. Answering the old unanswered questions will be most of the part meaningless as most of the people looking for code review would have moved on and unlike stackoverflow don't have a lot of value unless the questions/code is a general programming problem. – Aseem Bansal Nov 16 '13 at 14:37
• @AseemBansal: I think answering them is the main intent. But an unanswered question should still be closed or deleted if necessary. There are still a lot of untouched questions, so you never know what you'll find. If you're unsure about one, flag it. – Jamal Nov 17 '13 at 7:23
• @AseemBansal I agree with Jamal. Zombies must be exterminated. It gives site regulars an opportunity to foster reviewing skills in preparation of the Big D-Day, and it turns out our answered% is now rated Excellent at 90%. This CR-review has awakened the Review Monster, let's see what comes out of it. – Mathieu Guindon Nov 24 '13 at 4:24
• Check out the 100-day update! for details on how the landscape has changed since November – rolfl Feb 26 '14 at 19:20

We did host our own election, just to let SE know that we are serious about our site, granted neither of the parties has yet reached Trusted user Status, but both are engaged in day to day activities on the site.

Code Review is in need of a new moderator

I know that we haven't graduated yet, but we wanted SE to know that we are thinking about the future of the site. We are preparing ourselves for elections, and other things that will come our way in terms of a graduated site.

I have also noticed one user that has risen past the 1.5k rep mark in less than 2 months here on the site, obviously answering a lot of questions, with really good answers.

I have also noticed that some users have actually come back to Code Review after not participating for a long period of time, and those users reputation has risen fairly quickly as well.

As far as the unanswered questions, I have been looking at some of the oldest questions that have no votes, and some of them were off topic, but most of them were not properly tagged. and yet others have huge blocks of code that take a while to actually read through, let alone give a reasonable answer. That is only in the tags that I work in most.

## Not all of our answers have to hit everything that is wrong with the code

This means that we can/should leave shorter answers even if the question is begging for more review, a lot of us are reviewing in between writing code and waiting for code to compile or whatever, and leaving shorter answers should (IS) okay. this will also leave room for allowing ( asking ) the OP to vote on all the answers that gave a good review to their code.

## We can post more than one answer on a question

I think this would give us more voting opportunities (up/down), because it would allow us, and other users, to pick the best reviews on the question and up vote them, and down vote the answers that are very low quality or don't fit the code/question.

## For Code Review where the code looks pretty good

We can also answer these questions with what the Code does well, as opposed to what needs to be changed, this is still a review. We do need to make sure that we explain why it is good coding practice or why it is efficient, so that it is a good review.

• +1 good answer. Definitely some things to consider for the future. Regarding moderators, I think they may be contacting the next potential one at this time as well. :-) We'll see. – Jamal Nov 15 '13 at 18:36
• @Jamal: it sounded to me like they were looking outside the box if you know what I mean? – Malachi Nov 15 '13 at 18:37
• You mean the aspects they were monitoring? Yeah, there are quite a bit. But at least we know what we need to do to improve. We just have to put it into practice. – Jamal Nov 15 '13 at 18:41
• @Jamal I think he meant InvalidCastException ;) – Mathieu Guindon Nov 15 '13 at 18:42
• @retailcoder: Or maybe it was just about boxes? :-P I've just woken up, so... – Jamal Nov 15 '13 at 18:44
• I mean, if they're already doing this review, then chances are they're looking for a new moderator as well. It's all private of course, so all we can do is keep checking the moderator section of the Users tab for an update. – Jamal Nov 15 '13 at 18:56
• @Jamal (and Malachi): we are in the process of filling the pro-tem moderator position. Since these are appointed by the Community Team, it's a little less public than usual. We don't want to put people in the position of publicly declining the job if they have private reservations. Having a meta post with pre-vetted candidates makes our job a lot easier however. (No guarantee that a candidate will be drawn from that pool, of course. It's just a useful place to start looking.) – Jon Ericson Nov 15 '13 at 20:55
• @JonEricson: I understand, and thank you for addressing that. I've already stated that this "election" is entirely unofficial, but I've also wanted to aid in this process by putting myself out there. Although obviously the community can put in their votes/thoughts/questions, I was mostly doing it for the powers that be. Regardless of what the community thinks, I too agree that this does help SE in making their decision. If anything, I also see this as an "insight" of the official election after graduation. – Jamal Nov 15 '13 at 20:59
• RE: "Not all of our answers have to hit everything that is wrong with the code." This was something that I didn't grok when I looked over the site last week. I really wanted to provide an answer or two, but the urge to provide a comprehensive answer (or none at all) was overwhelming. It might be helpful if site regulars led by example and started answering unanswered (meaning no answers at all) in this way. (N.B. I looked at Perl and Lua questions, which are probably minorities here. That might be a different problem than what one might see on more popular tags.) – Jon Ericson Nov 15 '13 at 21:01
• @Dagg: This is a classic chicken and egg problem. Until a user comes along who can review Lua, lua questions will look unloved, which will discourage both askers and answerers. Why not try out a self-answer? (Not that I know how a self-answer would work. Perhaps you could go back to that old question and see if you can apply what you know now to that old code?) – Jon Ericson Nov 15 '13 at 23:42
• @Jamal, it's not so much that my new code is a "viable answer," it's just a few improvements to the original code. I would have upvoted it as an answer, but probably not accepted it, because what I'm really looking for is an answer that can either reassure me that my approach is a good one or convince me that it's not. – Dagg Nov 15 '13 at 23:54
• @Dagg, I am saying there aren't many people reviewing code for Lua, leave an answer for your question, don't worry about the upvotes or the accepting the answer part. the more visibility there is for lua Q&A on this site, the more people will come that write lua code, the more chance you have of getting the review that you want. even questions with 1+ answers show up on the unanswered list if none of the answers are accepted – Malachi Nov 16 '13 at 6:04
• @Dagg: on your Question it looks like you have 4 up votes and that the people that know lua think that there isn't anything to review, if you have something to improve that code, post it. I think you are doing something right if they upvoted and didn't provide any reviews. – Malachi Nov 16 '13 at 6:35
• @Malachi: I can understand that, but it also bothers me. Even if that's the case, the system will still consider it unanswered. There's also no sense in closing it because the question should still receive answers. I suppose the best we can do is determine which unanswered questions do not apply to this, and try to get them answered. We can afford a few percentage points below 100%. – Jamal Nov 17 '13 at 7:29
• @Jamal, I was trying to talk Dagg into answering his own question, as he said he had made improvements to his code but it wasn't what he was looking for. – Malachi Nov 17 '13 at 7:33

I can see two kinds of questions where it is also difficult to provide an answer:

• Sometimes, you just want to say "rewrite everything from scratch" and reviewing the code would mean providing hints to rewrite the code using another paradigm. The main problem is that we don't know beforehand whether the user knows the said programming paradigm. It's not simply telling what would be better, but really trying to teach how to use yet another programming paradigm to the OP; that means a huge answer somehow not related to "reviewing" but which looks like a tutorial on how to program. I have problems putting words on what I want to say there, I hope you understand.
• There are people who post short pieces of code in which nobody could actually find any issue. It seems that, provided they are simple enough, those pieces of code are "perfect": good names, good coding style, correct use of idiomatic ways to solve the problem they tried to solve... Well, there is simply nothing to review anymore. And nobody will post "Well, this piece of code is just perfect. Nothing else to say." as an answer; it would only be a comment and this would become yet another answer.

Also, another problem that I saw from time to time is people answering "What you are trying to code already exists in some library; use that library". While it may help on StackOverflow, it does not help on CodeReview where lots of people post code written for learning purposes. But it seems that many users are somehow reluctant to provide an answer after having read such comments.

Well, otherwise, I totally agree with what has been said in the fist post :)

• leaving an answer that describes what they did right is a valid code review as long as you explain what they did right. – Malachi Nov 15 '13 at 14:44
• +1 about people saying that functionality already exists in some library. Those answers should be made invalid actually because the code has been posted for learning purposes many times. – Aseem Bansal Nov 16 '13 at 14:32
• @AseemBansal: Agreed. Should we start using the [reinventing-the-wheel] tag more often, then? – Jamal Nov 16 '13 at 18:28
• @Jamal I didn't know that such a tag existed. I'll definitely suggest using that. – Aseem Bansal Nov 16 '13 at 20:06

I'm very new here, but I'll say what I think anyway.

I think we need to be explicit about partial code reviews being ok. I think we ought to start putting things like "This is a partial review focusing on good variable names" at the top of answers. That way, people who are new to the site see partial reviews being done and don't have to worry about (or ask) whether that sort of thing is ok around here.

Same thing with upvotes. I haven't been here long, and I've felt like there aren't enough votes around here, but looking at my stats, I've only upvoted twice. I've been on Programmers longer, but I've got 201 votes over there, and it feels like a smaller thing to upvote there than here. An upvote shouldn't feel like a big deal. It should seem like the sort of thing people do around here, not an unusual thing for special circumstances.

Let me put it this way -- if I went through all of the questions on the front page, and upvoted all of them, the number of upvotes on the front page would more than double.

As far as the backlog goes, I think that's a bit of an artificial problem. The OP's of the 2 year old unanswered questions are long gone. They aren't there to edit their questions so the questions aren't the code equivalent of a wall of text, or to clarify anything. They aren't going to be there to learn anything, and they might not need to learn from the problems in their old code because they've already learned about it from elsewhere.

It would be worth reviewing code even when the OP almost certainly won't see the review if it were probable that someone else might read the review. If I'm pretty sure that reviewing an old unanswered question will result in me spending an hour of my time writing something that nobody else will read, I'm not likely to do it. I'd rather get a downvote and an explanation of why I'm dumb than have nobody read an answer.

I think worrying about not letting recent unanswered questions slip through the cracks would be useful, but I see a lot less benefit in worrying about things that have long since slipped through the cracks. On a lot of stackexchange sites, answering a question long after the OP has vanished makes sense, because other people are likely to want to ask the same question and benefit from a good answer. I don't think it makes as much sense here.

If reviewers could be reasonably sure of getting an upvote or two (or maybe even a downvote) for a review that doesn't cover everything, they'd have a decent motive to spend time on a question even when the OP is long gone.

How should we handle questions like this? Re-answer them for no good reason, hoping to get our numbers up? Leave them alone? Close them?

• Check out some of these Meta Q&A's link another link – Malachi Nov 18 '13 at 15:13
• As I've heard before, answering in the comments is not encouraged. However, I understand some people are hesitant about putting out a full answer, and I can be this way sometimes (at least on SO). For here, these answers in the comments are only important if they're on-topic analyses. If they're bug fixes, then that's okay (well, still off-topic, but it's better to get the OP's code working somehow). However, I don't feel right giving answers for such on-topic comments. That may just be me, though. I suppose if it's for an old question (users are away), then it may still be okay. – Jamal Nov 19 '13 at 0:06

Just a little thing about codereview's reach. I don't have enough statistics about which rep user visited when but I have been collecting the statistics of visitors from front page for past 2 months and have come to the conclusion that codereview's reach is gradually increasing. Here's a graph using python/matplotlib.

EDIT

Ok now I'll post an image about the number of questions and answers which is a little worrying.

• The green line is the answers
• blue line below is the number of questions.

Doing a little maths I saw that there is a problem. The slope of both has remained nearly the same from the time I started collecting the data. The ratio is not changing. We are stuck with a ratio of 1.68 answers per question.

EDIT 2

I stopped collecting the data at the beginning of feb but here's what I have.

• Getting the unanswered questions answered will pull that ratio up. – Mathieu Guindon Nov 17 '13 at 15:51
• I'd be curious to see what it looks like with December 2013 and January 2014 in the picture :) – Mathieu Guindon Feb 19 '14 at 3:32
• @Mat'sMug Added the screenshot. – Aseem Bansal Feb 19 '14 at 15:27

Regarding user retention… it seems that on almost every StackExchange site, only one quarter of the users ever post more than once or twice. Does Code Review fit that pattern? If so, does user retention matter much?

Here's my methodology:

1. For all users of a site who have asked or answered anything, count their questions and answers.
2. For each user, the number of questions + answers is their "activity".
3. Categorize users according to their quintile of activity.
4. For each quintile of users, get the average reputation, the average proportion of posts that are questions, and the average proportion of posts that are answers.

Compare the results:

• Code Review: 4th quintile has 1.8 posts per user, 5th quintile has 9.1 posts per user.
• Web Applications: 4th quintile has 1.4 posts per user, 5th quintile has 9.2 posts per user.
• Programmers: 4th quintile has 2.4 posts per user, 5th quintile has 21.0 posts per user.
• Server Fault: 4th quintile has 2.5 posts per user, 5th quintile has 22.5 posts per user.
• SuperUser: 4th quintile has 2.0 posts per user, 5th quintile has 17.2 posts per user.
• Ask Ubuntu: 4th quintile has 1.4 posts per user, 5th quintile has 11.0 posts per user.
• English: 4th quintile has 1.9 posts per user, 5th quintile has 22.7 posts per user.

The pattern is consistent: nearly all the activity comes from the top quartile of users, and the bulk of the users are drive-bys. There are some minor differences — for example, the top quintile of Programmers users skew towards answering, and the the bottom half of Ask Ubuntu users skew towards asking.

I'd like to point out that the statistics for Code Review are very similar to Web Applications, a graduated site, in other ways as well:

• Questions — CR: 11k, WA: 14k
• Answers — CR: 19k, WA: 21k
• Answered — CR: 93%, WA: 80%
• Users — CR: 32k, WA: 39k
• Reputation range of first page of users — CR: 2.5k–17.9k, WA: 2.4k–15.4k
• Regarding the Data Explorer, you may have to look into that on MSO. Propose a feature-request if necessary, only if there aren't already sufficient posts on it. – Jamal Nov 19 '13 at 1:09
• Feature request on MSO: meta.stackexchange.com/a/207961/148099 — Please upvote! – 200_success Nov 19 '13 at 1:16
• How do you find the name of the tables on which to write queries? – Aseem Bansal Nov 19 '13 at 15:00
• @AseemBansal When you compose the query, the sidebar shows the available tables and columns. – 200_success Nov 19 '13 at 15:54
• I am going to have to check this out later. I like the idea of writing some queries to show data on our site – Malachi Nov 19 '13 at 16:34
• you should post this as a feature request, so that everyone can vote on it and we can flag it for moderator attention – Malachi Nov 19 '13 at 16:35
• I'll ask about how we deal with Data Explorer but for now I'll run a copy of your query on our end and let you know how things come out for you. – Grace Note Nov 25 '13 at 5:31
• Alright, here's some wordage on the analysis. Chat seemed a bit better of a vehicle than trying to do a bunch of comments. – Grace Note Nov 25 '13 at 6:46
• Back in November, I think I had something like 550 rep score. Things changed, and fast. At least the first users page did. – Mathieu Guindon Jan 16 '14 at 0:14

It seems that many people are simply unaware of this site. I've never seen it mentioned on SO and only discovered it by accident from a moderator's comment on my question. Maybe raising awareness is what can solve other problems?