How is Code Review doing right now?

Question

I’m Grace Note, a Community Manager at Stack Exchange. Normally for most sites we run a community site evaluation, as explained in this Meta Stack Overflow post under Public Beta sites. However, these reviews involve comparing our site content against searches on the internet for competing answers - which doesn’t entirely mesh with what we do on this particular site. For this reason we haven’t been running any community site evaluations on Code Review, and instead perform internal reviews on our own periodic basis. We recently completed one such review and so here’s some feedback for y’all.

Voting and Reputation Issues

There’s a bit of an issue with voting, and by consequence reputation on the site. Code Golf made a comparison themselves and figured that a major problem was in the lack of reputation by virtue of the lack of voting. Code Review is doing a fair amount better in that department (having 3 users at over 10k) and having a decent spread (21 users can vote to close/reopen, and all 40 of the first page can edit anything), but it dwindles quickly after that. 40 looks cool but the number of avid users for the site is almost a thousand, rendering that as less than 4% of the “active userbase” (more on that later) with decent privileges.

Voting is an odd beast here. When we reviewed the site, the front page had a dearth of voting. Even looking now, in the last 100 questions asked and the last 100 answers posted, less than 10 of each have scores past 3 with several negatively rated. Now, there are users who vote - vote a lot in fact, as most of the top users have hundreds of votes under their belts (with decent downvote rates as well). But in spite of this, there’s still a humongous bed of users sitting in the ~100-200 reputation range and not actually in the major privilege levels. So for where-ever all the votes are going, it ain’t accumulating.

Community Engagement

One positive point is that this site has an extremely engaged Meta. There’s healthy activity in recent days that surpasses a lot of other sites that we check up for graduation review.

For all that engagement, though, the site itself isn’t keeping people. User retention is less than ideal - there’s almost a thousand users listed as “avid” on Area 51, but according to our traffic metrics less than a hundred of them visit on a regular basis, and less than two hundred were even seen in the last month. These aren’t happy numbers. They are in fact unhappy numbers. Very unhappy numbers.

The majority of this is in the lower reputation brackets. The top reputation users consistently visit the site but as we break down below 2000 and 1000 reputation, the retention rate drops exponentially. By 1000 we’re already at half, and again we lose almost 90% by even just the 150 reputation range. This basically appears to show that people who already have an investment here are sticking around, but few new people truly start big journeys.

Time to Answer

Jon Ericson, another Community Manager, ran some numbers on the site and discovered that the average time it takes for a question to receive its first answer is actually several hours (averaging somewhere between 8 and 10 hours). This seemed pretty high, but a quick browse of the questions is pretty enlightening - things are huge here. A lot of questions comprise of gigantic blocks of code that have little guidance as to what should be reviewed. This, however, can be remedied a bit.

• Ask people to provide less gigantic blocks of code and instead scope their code to the relevantly smaller portions
• Ask people to provide instructions and/or guidance as to what they’re looking to get out of the review or what they want to have addressed
• Encourage a stronger positive feedback loop such as by editing

Monstrous code blocks are a two-fold to the site’s potential. It scares away new answerers because of the amount of effort needed to get in (which, with voting distribution, isn’t necessarily rewarding either). And without getting answerers, askers don’t get as much out of the site leaving them to vanish as well. Which brings me to the final point...

Unanswered

Tying in to all of the above is one of the most pressing concerns about the site I had in my own review, which is the gigantic pool of unanswered questions. There are, at the time of this writing, 1237 questions without any upvoted answers, a whopping 1010 of which have absolutely no answers at all. That’s over 10% of the site’s questions, sitting untouched and barely considered to be alive. These questions range from the site’s birth in 2011 all the way to modern day 2013. About 25% of these questions result in users never returning after a week, from a rudimentary query.

This isn’t a small number. This is a huge chunk of the site that’s basically just a black hole of despair, swallowing everything without giving anything back. And there’s a simple enough fix to this - clean up the unanswered questions. A large amount of what I see from these is questions that get a fair amount of comments back-and-forth when first asked, but then the entire dialogue is dropped and no one on either side follows up. The questions essentially become abandoned, not unlike the ultimate fate of this site if people don’t keep up.

---

In conclusion, the main problem points are the distribution of votes and reputation resulting in a largely bottom-heavy reputation spread that cannot support the site, an exceedingly long average time between answers, and severe question abandonment issues resulting in a lot of unanswered questions now and old.

Past that, though, the site’s actually doing rather well. It has a humongous userbase behind it (albeit, again, bottom-heavy), the userbase is engaged on Meta, traffic metrics are superb, and overall the site has a positive feel. When we’re not looking at the stagnant and decayed questions, I see extremely in-depth reviews and helpful advice, as well as both askers and answerers working together. However, between the lack of voting and the poor question/answer return rate, it isn’t ready to see graduation.

I hope y’all find our findings to be helpful, and let us know if you have any further questions. For example if you yourselves know any particular fall points that you’d like advice on, let us know.

If you've made it this far in here, check out the 100 days update that y'all fished up! It's worth a study and comparison!

Answer 1

We did host our own election, just to let SE know that we are serious about our site, granted neither of the parties has yet reached Trusted user Status, but both are engaged in day to day activities on the site.

Code Review is in need of a new moderator

I know that we haven't graduated yet, but we wanted SE to know that we are thinking about the future of the site. We are preparing ourselves for elections, and other things that will come our way in terms of a graduated site.

---

I have also noticed one user that has risen past the 1.5k rep mark in less than 2 months here on the site, obviously answering a lot of questions, with really good answers.

I have also noticed that some users have actually come back to Code Review after not participating for a long period of time, and those users reputation has risen fairly quickly as well.

As far as the unanswered questions, I have been looking at some of the oldest questions that have no votes, and some of them were off topic, but most of them were not properly tagged. and yet others have huge blocks of code that take a while to actually read through, let alone give a reasonable answer. That is only in the tags that I work in most.

---

Note to other users on the site

Not all of our answers have to hit everything that is wrong with the code

This means that we can/should leave shorter answers even if the question is begging for more review, a lot of us are reviewing in between writing code and waiting for code to compile or whatever, and leaving shorter answers should (IS) okay. this will also leave room for allowing ( asking ) the OP to vote on all the answers that gave a good review to their code.

We can post more than one answer on a question

I think this would give us more voting opportunities (up/down), because it would allow us, and other users, to pick the best reviews on the question and up vote them, and down vote the answers that are very low quality or don't fit the code/question.

For Code Review where the code looks pretty good

We can also answer these questions with what the Code does well, as opposed to what needs to be changed, this is still a review. We do need to make sure that we explain why it is good coding practice or why it is efficient, so that it is a good review.

Answer 2

I can see two kinds of questions where it is also difficult to provide an answer:

• Sometimes, you just want to say "rewrite everything from scratch" and reviewing the code would mean providing hints to rewrite the code using another paradigm. The main problem is that we don't know beforehand whether the user knows the said programming paradigm. It's not simply telling what would be better, but really trying to teach how to use yet another programming paradigm to the OP; that means a huge answer somehow not related to "reviewing" but which looks like a tutorial on how to program. I have problems putting words on what I want to say there, I hope you understand.
• There are people who post short pieces of code in which nobody could actually find any issue. It seems that, provided they are simple enough, those pieces of code are "perfect": good names, good coding style, correct use of idiomatic ways to solve the problem they tried to solve... Well, there is simply nothing to review anymore. And nobody will post "Well, this piece of code is just perfect. Nothing else to say." as an answer; it would only be a comment and this would become yet another answer.

Also, another problem that I saw from time to time is people answering "What you are trying to code already exists in some library; use that library". While it may help on StackOverflow, it does not help on CodeReview where lots of people post code written for learning purposes. But it seems that many users are somehow reluctant to provide an answer after having read such comments.

Well, otherwise, I totally agree with what has been said in the fist post :)

Answer 3

I'm very new here, but I'll say what I think anyway.

I think we need to be explicit about partial code reviews being ok. I think we ought to start putting things like "This is a partial review focusing on good variable names" at the top of answers. That way, people who are new to the site see partial reviews being done and don't have to worry about (or ask) whether that sort of thing is ok around here.

Same thing with upvotes. I haven't been here long, and I've felt like there aren't enough votes around here, but looking at my stats, I've only upvoted twice. I've been on Programmers longer, but I've got 201 votes over there, and it feels like a smaller thing to upvote there than here. An upvote shouldn't feel like a big deal. It should seem like the sort of thing people do around here, not an unusual thing for special circumstances.

Let me put it this way -- if I went through all of the questions on the front page, and upvoted all of them, the number of upvotes on the front page would more than double.

As far as the backlog goes, I think that's a bit of an artificial problem. The OP's of the 2 year old unanswered questions are long gone. They aren't there to edit their questions so the questions aren't the code equivalent of a wall of text, or to clarify anything. They aren't going to be there to learn anything, and they might not need to learn from the problems in their old code because they've already learned about it from elsewhere.

It would be worth reviewing code even when the OP almost certainly won't see the review if it were probable that someone else might read the review. If I'm pretty sure that reviewing an old unanswered question will result in me spending an hour of my time writing something that nobody else will read, I'm not likely to do it. I'd rather get a downvote and an explanation of why I'm dumb than have nobody read an answer.

I think worrying about not letting recent unanswered questions slip through the cracks would be useful, but I see a lot less benefit in worrying about things that have long since slipped through the cracks. On a lot of stackexchange sites, answering a question long after the OP has vanished makes sense, because other people are likely to want to ask the same question and benefit from a good answer. I don't think it makes as much sense here.

If reviewers could be reasonably sure of getting an upvote or two (or maybe even a downvote) for a review that doesn't cover everything, they'd have a decent motive to spend time on a question even when the OP is long gone.

EDIT: I've looked at a couple of old/unanswered questions and there's something I've seen a few times that I think is pretty common: people answering in the comments. This question for example. The post has no answers, but it has helpful comments, and all of the helpful comments (which the OP noted and appreciated) were within 32 minutes of the question being posted. By the numbers, we haven't answered this question at all, but in reality, it was answered helpfully less than an hour after it was posted.

How should we handle questions like this? Re-answer them for no good reason, hoping to get our numbers up? Leave them alone? Close them?

Answer 4

Just a little thing about codereview's reach. I don't have enough statistics about which rep user visited when but I have been collecting the statistics of visitors from front page for past 2 months and have come to the conclusion that codereview's reach is gradually increasing. Here's a graph using python/matplotlib.

EDIT

Ok now I'll post an image about the number of questions and answers which is a little worrying.

• The green line is the answers
• blue line below is the number of questions.

Doing a little maths I saw that there is a problem. The slope of both has remained nearly the same from the time I started collecting the data. The ratio is not changing. We are stuck with a ratio of 1.68 answers per question.

EDIT 2

I stopped collecting the data at the beginning of feb but here's what I have.

Answer 5

Regarding user retention… it seems that on almost every StackExchange site, only one quarter of the users ever post more than once or twice. Does Code Review fit that pattern? If so, does user retention matter much?

Here's my methodology:

1. For all users of a site who have asked or answered anything, count their questions and answers.
2. For each user, the number of questions + answers is their "activity".
3. Categorize users according to their quintile of activity.
4. For each quintile of users, get the average reputation, the average proportion of posts that are questions, and the average proportion of posts that are answers.

Compare the results:

• Code Review: 4th quintile has 1.8 posts per user, 5th quintile has 9.1 posts per user.
• Web Applications: 4th quintile has 1.4 posts per user, 5th quintile has 9.2 posts per user.
• Programmers: 4th quintile has 2.4 posts per user, 5th quintile has 21.0 posts per user.
• Server Fault: 4th quintile has 2.5 posts per user, 5th quintile has 22.5 posts per user.
• SuperUser: 4th quintile has 2.0 posts per user, 5th quintile has 17.2 posts per user.
• Ask Ubuntu: 4th quintile has 1.4 posts per user, 5th quintile has 11.0 posts per user.
• English: 4th quintile has 1.9 posts per user, 5th quintile has 22.7 posts per user.

The pattern is consistent: nearly all the activity comes from the top quartile of users, and the bulk of the users are drive-bys. There are some minor differences — for example, the top quintile of Programmers users skew towards answering, and the the bottom half of Ask Ubuntu users skew towards asking.

I'd like to point out that the statistics for Code Review are very similar to Web Applications, a graduated site, in other ways as well:

• Questions — CR: 11k, WA: 14k
• Answers — CR: 19k, WA: 21k
• Answered — CR: 93%, WA: 80%
• Users — CR: 32k, WA: 39k
• Reputation range of first page of users — CR: 2.5k–17.9k, WA: 2.4k–15.4k

Answer 6

It seems that many people are simply unaware of this site. I've never seen it mentioned on SO and only discovered it by accident from a moderator's comment on my question. Maybe raising awareness is what can solve other problems?