I’m Grace Note, a Community Manager at Stack Exchange. Normally for most sites we run a community site evaluation, as explained in this Meta Stack Overflow post under Public Beta sites. However, these reviews involve comparing our site content against searches on the internet for competing answers - which doesn’t entirely mesh with what we do on this particular site. For this reason we haven’t been running any community site evaluations on Code Review, and instead perform internal reviews on our own periodic basis. We recently completed one such review and so here’s some feedback for y’all.
Voting and Reputation Issues
There’s a bit of an issue with voting, and by consequence reputation on the site. Code Golf made a comparison themselves and figured that a major problem was in the lack of reputation by virtue of the lack of voting. Code Review is doing a fair amount better in that department (having 3 users at over 10k) and having a decent spread (21 users can vote to close/reopen, and all 40 of the first page can edit anything), but it dwindles quickly after that. 40 looks cool but the number of avid users for the site is almost a thousand, rendering that as less than 4% of the “active userbase” (more on that later) with decent privileges.
Voting is an odd beast here. When we reviewed the site, the front page had a dearth of voting. Even looking now, in the last 100 questions asked and the last 100 answers posted, less than 10 of each have scores past 3 with several negatively rated. Now, there are users who vote - vote a lot in fact, as most of the top users have hundreds of votes under their belts (with decent downvote rates as well). But in spite of this, there’s still a humongous bed of users sitting in the ~100-200 reputation range and not actually in the major privilege levels. So for where-ever all the votes are going, it ain’t accumulating.
One positive point is that this site has an extremely engaged Meta. There’s healthy activity in recent days that surpasses a lot of other sites that we check up for graduation review.
For all that engagement, though, the site itself isn’t keeping people. User retention is less than ideal - there’s almost a thousand users listed as “avid” on Area 51, but according to our traffic metrics less than a hundred of them visit on a regular basis, and less than two hundred were even seen in the last month. These aren’t happy numbers. They are in fact unhappy numbers. Very unhappy numbers.
The majority of this is in the lower reputation brackets. The top reputation users consistently visit the site but as we break down below 2000 and 1000 reputation, the retention rate drops exponentially. By 1000 we’re already at half, and again we lose almost 90% by even just the 150 reputation range. This basically appears to show that people who already have an investment here are sticking around, but few new people truly start big journeys.
Time to Answer
Jon Ericson, another Community Manager, ran some numbers on the site and discovered that the average time it takes for a question to receive its first answer is actually several hours (averaging somewhere between 8 and 10 hours). This seemed pretty high, but a quick browse of the questions is pretty enlightening - things are huge here. A lot of questions comprise of gigantic blocks of code that have little guidance as to what should be reviewed. This, however, can be remedied a bit.
- Ask people to provide less gigantic blocks of code and instead scope their code to the relevantly smaller portions
- Ask people to provide instructions and/or guidance as to what they’re looking to get out of the review or what they want to have addressed
- Encourage a stronger positive feedback loop such as by editing
Monstrous code blocks are a two-fold to the site’s potential. It scares away new answerers because of the amount of effort needed to get in (which, with voting distribution, isn’t necessarily rewarding either). And without getting answerers, askers don’t get as much out of the site leaving them to vanish as well. Which brings me to the final point...
Tying in to all of the above is one of the most pressing concerns about the site I had in my own review, which is the gigantic pool of unanswered questions. There are, at the time of this writing, 1237 questions without any upvoted answers, a whopping 1010 of which have absolutely no answers at all. That’s over 10% of the site’s questions, sitting untouched and barely considered to be alive. These questions range from the site’s birth in 2011 all the way to modern day 2013. About 25% of these questions result in users never returning after a week, from a rudimentary query.
This isn’t a small number. This is a huge chunk of the site that’s basically just a black hole of despair, swallowing everything without giving anything back. And there’s a simple enough fix to this - clean up the unanswered questions. A large amount of what I see from these is questions that get a fair amount of comments back-and-forth when first asked, but then the entire dialogue is dropped and no one on either side follows up. The questions essentially become abandoned, not unlike the ultimate fate of this site if people don’t keep up.
In conclusion, the main problem points are the distribution of votes and reputation resulting in a largely bottom-heavy reputation spread that cannot support the site, an exceedingly long average time between answers, and severe question abandonment issues resulting in a lot of unanswered questions now and old.
Past that, though, the site’s actually doing rather well. It has a humongous userbase behind it (albeit, again, bottom-heavy), the userbase is engaged on Meta, traffic metrics are superb, and overall the site has a positive feel. When we’re not looking at the stagnant and decayed questions, I see extremely in-depth reviews and helpful advice, as well as both askers and answerers working together. However, between the lack of voting and the poor question/answer return rate, it isn’t ready to see graduation.
I hope y’all find our findings to be helpful, and let us know if you have any further questions. For example if you yourselves know any particular fall points that you’d like advice on, let us know.
If you've made it this far in here, check out the 100 days update that y'all fished up! It's worth a study and comparison!