On the huge amount of work needed for a CR answer

Question

I'm a regular at Stack Overflow, and I just recently discovered Code Review, where I answered some questions with the d3.js tag. We all know that CR is quite different from SO: by its very nature, both the questions and answers at CR are huge, full of code and explanations. I can't imagine a one-line answer here at CR, although they are quite common (and sometimes heavily upvoted) at SO.

We are programmers trying to help fellow programmers, that's the most important part. However, those green points we (sometimes) receive are an important part of the game, even if most people deny it. And that's why I'm writing this question, which is not exactly a question but a discussion starter instead.

It seems to me that there is a big disproportion between the amount of work we have answering a question at CR and the number of votes we (may) get. For instance, in my case, I wrote 7 answers, totalling around 5000 words. Just for having a better picture of what that means, I decided to pick 7 answers of mine at SO, randomly (I know this lacks scientific validity, it's just for illustrate the issue). Then, comparing both sets of answers, I analysed the number of words and the number of votes in each, creating this bar chart (as a bonus, if you hover over the bars, you'll get a nice red freehand!):

var w=600,h=250,padding=[10,10,100,80],data=[{site:"Stack Overflow",words:1121,votes:22},{site:"Code Review",words:5083,votes:13}],svg=d3.select("body").append("svg").attr("width",w).attr("height",h),xScale=d3.scaleLinear().range([padding[3],w-padding[1]]).domain([0,1.1*d3.max(data,function(t){return t.words/t.votes})]),yScale=d3.scaleBand().range([padding[0],h-padding[2]]).domain(data.map(function(t){return t.site})).padding(.5),colorScale=d3.scaleOrdinal(["#ff7f0e","#1f77b4"]),bars=svg.selectAll(null).data(data).enter().append("rect").style("fill",function(t,e){return colorScale(e)}).attr("x",padding[3]).attr("width",0).attr("y",function(t){return yScale(t.site)}).attr("height",yScale.bandwidth());bars.transition().duration(3e3).attr("width",function(t){return xScale(t.words/t.votes)-padding[3]});var labels=svg.selectAll(null).data(data).enter().append("text").attr("dominant-baseline","central").attr("y",function(t){return yScale(t.site)+yScale.bandwidth()/2}).attr("x",padding[3]+10).style("font-family","sans-serif").style("font-size","12px");labels.transition().duration(3e3).attr("x",function(t){return xScale(t.words/t.votes)+10}).attrTween("text",function(t){var e=this,a=d3.interpolateNumber(0,t.words/t.votes);return function(t){return d3.select(e).text(~~a(t))}});var freehandCoord="m -2.0278636,-4.2995802 c -0.4608758,0.1152189 -0.6108077,0.047127 -0.8690844,0.4345422 -0.026782,0.040174 0.040173,0.1180651 0,0.1448474 -0.1270396,0.084693 -0.2979788,0.076566 -0.4345422,0.1448474 -0.061073,0.030536 -0.083774,0.1143109 -0.1448474,0.1448474 -0.043185,0.021593 -0.1107065,-0.034141 -0.1448474,0 -0.076341,0.076341 -0.08496,0.1998643 -0.1448474,0.2896948 -0.037876,0.056814 -0.1232548,0.08007 -0.1448474,0.1448474 -0.030537,0.09161 0.030537,0.1980853 0,0.2896948 -0.043185,0.1295555 -0.213943,0.1760672 -0.2896948,0.2896948 -0.059887,0.089831 -0.08496,0.1998643 -0.1448474,0.2896948 -0.037876,0.056814 -0.1143109,0.083774 -0.1448474,0.1448474 -0.021593,0.043185 0,0.096565 0,0.1448474 -0.048283,0.048283 -0.1143109,0.083774 -0.1448474,0.1448474 -0.021593,0.043185 0.034141,0.1107066 0,0.1448474 -0.034141,0.034141 -0.1232548,-0.043185 -0.1448474,0 -0.043185,0.08637 0.030537,0.1980853 0,0.2896948 -0.021593,0.064778 -0.1143109,0.083774 -0.1448474,0.1448474 -0.043185,0.08637 0.068282,0.2214131 0,0.28969482 -0.034141,0.0341409 -0.1107065,-0.0341409 -0.1448474,0 -0.034141,0.0341409 0,0.0965649 0,0.1448474 0,0.0482825 0,0.0965649 0,0.1448474 0,0.4345422 0,0.8690844 0,1.3036266 0,0.090693 -0.033333,0.80241788 0,0.86908438 0.030536,0.061073 0.1143109,0.083774 0.1448474,0.1448474 0.043185,0.08637 -0.043185,0.2033245 0,0.2896948 0.030537,0.061073 0.1232548,0.08007 0.1448474,0.1448474 0.043454,0.1303627 -0.043454,0.3041796 0,0.4345422 0.034141,0.1024226 0.1107065,0.1872722 0.1448474,0.2896948 0.030536,0.09161 -0.043185,0.2033245 0,0.2896948 0.030536,0.061073 0.1143109,0.083774 0.1448474,0.1448474 0.021593,0.043185 -0.034141,0.1107066 0,0.1448474 0.034141,0.034141 0.1107065,-0.034141 0.1448474,0 0.034141,0.034141 -0.021593,0.1016623 0,0.1448474 0.030537,0.061073 0.096565,0.096565 0.1448474,0.1448474 0.048283,0.048283 0.083774,0.1143109 0.1448474,0.1448474 0.043185,0.021593 0.1180651,-0.040173 0.1448474,0 0.084693,0.1270396 0.1078164,0.2864184 0.1448474,0.4345422 0.01171,0.046841 -0.043185,0.1232549 0,0.1448474 0.08637,0.043185 0.1950051,-0.018938 0.2896948,0 0.1497176,0.029943 0.2979788,0.076566 0.4345422,0.1448474 0.2477582,0.1238791 0.041937,0.3106631 0.2896948,0.4345422 0.1295554,0.064778 0.2971279,-0.045805 0.4345422,0 0.045805,0.015268 -0.043185,0.1232549 0,0.1448474 0.08637,0.043185 0.1980853,-0.030537 0.2896948,0 0.1295554,0.043185 0.1675487,0.2286218 0.2896948,0.2896948 0.08637,0.043185 0.2214131,-0.068282 0.2896948,0 0.034141,0.034141 -0.043185,0.1232549 0,0.1448474 0.1484787,0.074239 0.57491496,-0.074661 0.724237,0 0.061073,0.030537 0.0837744,0.1143109 0.1448474,0.1448474 0.0898886,0.044944 0.51495771,0 0.5793896,0 0.20716904,0 0.7186015,0.050161 0.8690844,0 0.10242258,-0.034141 0.1872722,-0.1107065 0.2896948,-0.1448474 0.1832191,-0.061073 0.3920261,0.046841 0.5793896,0 C 1.8044118,5.6783297 1.8262024,5.5879189 1.8830162,5.550043 2.1542787,5.3692014 2.3359907,5.4411899 2.6072532,5.2603482 2.8438191,5.1026376 2.666714,5.0456697 3.0417954,4.9706534 c 0.1420346,-0.028407 0.2940196,0.035131 0.4345422,0 C 3.5425806,4.9540927 3.5729025,4.8740885 3.621185,4.825806 3.7177499,4.7292411 3.7937776,4.6063725 3.9108798,4.5361112 4.041804,4.4575567 4.2088586,4.4595455 4.345422,4.3912638 4.406495,4.3607273 4.4334556,4.2842923 4.4902694,4.2464164 4.5801,4.1865294 4.6873868,4.1571155 4.7799642,4.101569 4.9292408,4.0120031 5.0588001,3.8897274 5.2145064,3.8118742 5.3510698,3.7435925 5.5269026,3.7586364 5.6490486,3.6670268 5.7354189,3.6022491 5.7175547,3.4536733 5.793896,3.377332 5.8702373,3.3009907 6.0072495,3.3088259 6.0835908,3.2324846 6.2846744,3.031401 6.6245186,1.9468578 6.6629804,1.6391632 c 0.023955,-0.1916385 -0.027313,-0.3882008 0,-0.5793896 0.2059204,-1.44144297 0.1448474,0.7600245 0.1448474,-1.01393178 0,-0.09656494 -0.043185,-0.2033245 0,-0.2896948 0.4180893,-0.83617862 0.2236664,0.33014192 0.4345422,-0.724237 0.018938,-0.0946897 0,-0.19312992 0,-0.28969482 0,-0.3379773 0,-0.6759545 0,-1.0139318 0,-0.1448474 0,-0.2896948 0,-0.4345422 0,-0.096565 0.035863,-0.2000365 0,-0.2896948 -0.064654,-0.1616338 -0.2250413,-0.2729084 -0.2896948,-0.4345422 -0.035863,-0.089658 0.018938,-0.1950051 0,-0.2896948 -0.029943,-0.1497176 -0.1078164,-0.2864183 -0.1448474,-0.4345422 -0.01171,-0.046841 0.015268,-0.099043 0,-0.1448474 C 6.6617671,-4.7377624 6.2844643,-4.9093739 5.9387434,-5.1686646 5.884118,-5.2096336 5.854969,-5.2829755 5.793896,-5.313512 c -0.043185,-0.021593 -0.1107065,0.034141 -0.1448474,0 C 4.6833993,-6.2791613 6.3732856,-4.9755347 5.2145064,-5.7480542 5.0136392,-5.8819657 4.8058211,-6.0118921 4.6351168,-6.1825964 4.4586951,-6.3590181 4.5672671,-6.5062161 4.345422,-6.6171386 4.2005746,-6.6654211 4.0557272,-6.7137035 3.9108798,-6.761986 3.8625968,-6.810269 3.8271058,-6.8762969 3.7660324,-6.9068334 c -0.043185,-0.021593 -0.1016622,0.021593 -0.1448474,0 -0.122146,-0.061073 -0.1601393,-0.2465096 -0.2896948,-0.2896948 -0.1832191,-0.061073 -0.3862597,0 -0.5793896,0 -0.4828247,0 -0.9656493,0 -1.448474,0 -0.048283,0 -0.096565,0 -0.1448474,0 -0.048283,0 -0.1107065,-0.034141 -0.1448474,0 -0.0341409,0.034141 0.034141,0.1107065 0,0.1448474 -0.0341409,0.034141 -0.11070654,-0.034141 -0.1448474,0 -0.0341409,0.034141 0.0215926,0.1016623 0,0.1448474 -0.0778531,0.1557063 -0.18524385,0.2952743 -0.2896948,0.4345422 0.014459,0.2869511 -0.21335352,0.068506 -0.2896948,0.1448474 -0.0341409,0.034141 0.0341409,0.1107065 0,0.1448474 -0.0341409,0.034141 -0.10166225,-0.021593 -0.1448474,0 C 0.08377437,-6.1520599 0.06107303,-6.0682855 0,-6.037749 c -0.04318515,0.021593 -0.11070654,-0.034141 -0.1448474,0 -0.0341409,0.034141 0.0341409,0.1107065 0,0.1448474 -0.0682817,0.068282 -0.22141308,-0.068282 -0.2896948,0 -0.0341409,0.034141 0.0341409,0.1107065 0,0.1448474 -1.3825e-4,1.383e-4 -0.43388569,-2.188e-4 -0.4345422,0 -0.0647777,0.021593 -0.0837744,0.1143109 -0.1448474,0.1448474 -0.043185,0.021593 -0.1107065,-0.034141 -0.1448474,0 -0.034141,0.034141 0.034141,0.1107065 0,0.1448474 -0.034141,0.034141 -0.1016623,-0.021593 -0.1448474,0 -0.061073,0.030537 -0.083774,0.1143109 -0.1448474,0.1448474 -0.043185,0.021593 -0.1107065,-0.034141 -0.1448474,0 -0.068282,0.068282 0.068282,0.2214131 0,0.2896948 -0.2286514,0.2286514 -0.088755,-0.3575027 -0.2896948,0.1448474 -0.073934,0.1848353 -0.096565,0.3862597 -0.1448474,0.5793896 -0.1448474,0.2414123 -0.3086369,0.4724265 -0.4345422,0.724237 -0.043185,0.08637 0.068282,0.2214131 0,0.2896948 -0.034141,0.034141 -0.1107065,-0.034141 -0.1448474,0 -0.034141,0.034141 0.021593,0.1016623 0,0.1448474 -0.030536,0.061073 -0.1143109,0.083774 -0.1448474,0.1448474 -0.021593,0.043185 0.034141,0.1107066 0,0.1448474 -0.1931299,0.1931299 -0.1448474,-0.2414123 -0.1448474,0.1448474",freehand=svg.append("path").attr("d",freehandCoord).style("fill","none").style("stroke","red").style("opacity",0).style("pointer-events","none"),xAxis=d3.axisBottom(xScale),yAxis=d3.axisLeft(yScale),gX=svg.append("g").attr("transform","translate(0,"+(h-padding[2])+")").call(xAxis),gY=svg.append("g").attr("transform","translate("+padding[3]+",0)").call(yAxis),xAxisText=svg.append("text").attr("x",padding[3]/2+(w-padding[1])/2).attr("text-anchor","middle").attr("y",h-padding[2]/2).text("Words per vote").style("font-family","sans-serif").style("font-size","12px");d3.timeout(function(){bars.on("mousemove",function(t,e){bars.style("opacity",function(t,a){return a===e?1:.25}),labels.style("opacity",function(t,a){return a===e?1:.25}),freehand.attr("transform","translate("+(xScale(t.words/t.votes)+10)+","+(yScale(t.site)+yScale.bandwidth()/2)+") scale(4)").style("opacity",1)}).on("mouseout",function(){labels.style("opacity",1),bars.style("opacity",1),freehand.style("opacity",0)})},3e3);

<script src="https://d3js.org/d3.v4.min.js"></script>

As we can see, supposing that this data is representative, one has to write almost 8x more words to get the same number of upvotes. I have to say that, for me, this is a reason for losing interest in participating at CR. Of course, we all know that the points system must be exactly the same in all SE sites, from SE Gardening and Landscaping to SE Worldbuilding.

So, this is my question: is there anything we can do? Or even better: should anything be done, or it's alright the way it is now?

Answer 1

I think it's unfair to Code Review to view reputation/vote as the metric for success of an answer or the real impact of your answer. The vote/reputation is a metric for Stack Exchange, but I feel here we're creating a bigger value. We're teaching people in a lot of ways, helping them and us at the same time. I know you probably know all of that, but my point is this is not taken into account.

one has to write almost 8x more words to get the same number of upvotes.

No we write 8x more words to help them get better, explain things and make sure they leave the question better than when they ask.

This is more important here than on Stack Overflow. I learned so much from post here! When I'm learning a new language I browse the tag here to see what are the common mistakes, best practices and small details about the language that you will not see in tutorials.

My post is a bit all over the place for explanation, but I just want to illustrate that you can't just look at votes and based the value of your contribution based on that.

Answer 2

IMO the answering scheme on SO is where the problem is. People answer quickies and get their upvotes, and walk away thinking they've made the Internet a better place.

Allow me to make a slight generalization here.

What I'm seeing on SO, from monitoring a specific tag on the daily, is a TON of similar questions involving the same handful of bugs, over and over. I finally got my gold badge on SO, so I can now hammer dupes... but a lot of "helpful" answers people put up on SO essentially boil down to this:

Try this:

{some code snippet}

And they get votes. Sure, it helped the OP. Sure, it might help someone that bothers searching for a solution to their problem... but it says what, not why, and doesn't explain how.

After spending quite some time reviewing code on CR, I've noticed a strange side-effect in my SO answering: I don't post quickies anymore. I just can't help explaining what's going on, why the compiler is throwing such or such error, how it's solved, and how to avoid it in the future... and it's not rare that I'll also point out problems the OP didn't ask about, but that are glaring in their code and just waiting to be the subject of their next SO question.

As a result, I end up spending anywhere from 15 to 45 minutes on a SO answer, which isn't too far off from the amount of effort I could put in a decent CR answer. And when I walk away I know I've made the Internet a better place.

So my conclusion is, CR isn't the problem. SO's voting culture is. I don't upvote "Try this: {code dump}" answers on SO. If people want a quick copy-pasta solution instead of actually understanding what's going on, it's their loss, and IMO that's not what SO was made for.

The best SO answers are the ones where the author clearly put a lot of time and effort. These are the answers that should be getting the votes. If they aren't, it's not CR's fault - it's SO's voting culture that's broken.

As for "words per vote"... as others have said, that's not a useful metric, specifically because you can post "helpful" answers that don't explain anything on SO and get votes for it.

Give a man a fish, and you feed him for a day; show him how to catch fish, and you feed him for a lifetime.

If SO's voting culture didn't value copy-pasta solutions that don't teach anything, your metrics would be quite different.

Answer 3

I think you'd be interested in Mat's Mug's answer to "What is the motivation to spend time doing a code review?":

I monitor and answer Stack Overflow questions in the vba tag - a tiny little tag in SO-land: ruby has 173,998 questions, vba has 73,666.

I'm tracking progress for my gold vba tag badge on SO.

I'm 124 answers above the criteria, yet still 138 votes short; that's 2.6 votes per answer. Contrast with Code Review, where I'm tracking the same badge:

That's 5.2 votes per answer, and a much more balanced progress (1000/200=5.0).

And so on CR it's likely that you get twice the upvotes per answer. (Still not scientific, but it's better than 7 questions)

I however disagree with you thinking words vs reputation gained are a good measure of whether it's worth answering. I spend about 30 minutes to 1 hour on most of my answers. Doesn't matter if it's one that has 30 words, or 1,000 words. Because not all of my time is spent writing. I spend time understanding the code and improving the code, both skills that come better the more I do it.

If you don't think spending however much time you do on your answers is worth it, then I'd encourage you to spend less time on your answers. IIRC we had a meta, that explicitly encouraged users to not try and improve everything. There was a time where three answers posted an answer to one question, and none of the answers overlapped, but all were improvements to the question.

Not only that but we allow iterative reviews. If you don't think you should write a long answer, but split your answer into two separate instances then you can hope the OP follows the advice in your first answer, and comes back to get the rest of your advice.

---

Bottom line, you don't have to spend 20 hours on an answer, stop when you think you've written enough.

Answer 4

I think the real question is whether there actually is a problem.

The answers are generally longer here, but I suspect the relative amount of high-quality posts is also higher. One-liner answers you can get away with on Stack Overflow are usually unacceptable here.

If there is a problem at all, I think it's high-quality posts not being rewarded enough. This can be (and sometimes is) in part fixed with a bounty.

On top of that there's the point that longer answers are not necessarily of higher quality than short posts. I tend to prefer longer answers though, and I know a couple of users who share that sentiment, so I applaud your efforts in this regard.

Thank you for keeping the quality of Code Review high.

Answer 5

There are many aspects that contribute to smaller reputation gains on CR compared to SO.

1. The word-count-to-vote ratio is a somewhat useful metric, but when we try to assess something in numbers, we definitely should not rely on a single indicator. For example, we could compare...

2. the head count difference. For the last seven days there were only 2'289 users visited CodeReview, whereas StackOverflow was visited by 659'153 users. That is already ~2.25 orders of magnitude in difference!

   Less eyes means less direct reputation effects. It also means less engagement which might spark conversation and more activity in terms of clicks/up votes. And that's just the registered users, which are only a part of visitors.

3. Speaking of the visitors, I also realized that the sites are being used in fundamentally different main ways.

   How does one get to StackOverflow? They normally search for an answer to a very specific question. Often, users get there directly from a search engine, they don't necessarily use the SO's search line...

   If same person is interested in a Code Review, I highly doubt he/she will even search for the question in Google. Instead, they'll come to CR.SE (if they are aware of it) and just post their question, because 99% of the time their code is unique enough. I'm intentionally dropping the custom-quicksearch-implementation question category since they are a minority.

   I believe this leads to questions that have thousand of up votes on StackOverflow. People there are luckier in a sense they can "invest into an answer once" and then keep receiving their dividends over the years! Here's an exaggerated example of such answer -- Jon Skeet has got 89k+ of reputation from up votes along (unless I misunderstand how reputation works with up votes). Now, I'm not saying that on SE you can answer one question and get tons of rep over time. No, for that you need to be very lucky to answer the "right" question in the "right" time. Yet, if you answer 100 questions, you'll see once in a while that you get +1 here or there and these reputation change do accumulate quite well.

   So, you can see the huge huge difference in ROI, because in CR.SE such investment is hardly even possible today. If somebody disagrees, ask them how often do they use the search bar in Code Review before posting their question.

4. UPDATE A graph of a function of active users, active voters, new question count, and average question votes would be a very interesting metric to look at.

5. Who knows what else? I'm very sure there are many more important aspects here...

---

This, however does not answer your question, right?

So, these measures may help to address the points from above:

1. You probably can not do much about word-to-count metric. At least, I don't see anything obvious here. If I have something to say, I say it, as long as it's important enough and useful.

2. Things will not change dramatically as long as this is a small community.

   Increase the head count -- bring the new users in (popularize & advertise?) and work on retention (be a welcoming community).

3. Provoke users to look into old questions. There are at least two built-in mechanisms already, but I'd argue their efficiency: a) Community user bumps the unanswered questions once in a while to the top of the page; b) there are "Linked" questions in the right hand side when you're inside a question, no idea how much attention it receives... If I am not mistaken, it is being populated with the questions that we refer to in our questions/answers, so we need to make sure we create this connections to older posts that are relevant to the context.

---

Just my 2 cents.

Oh, and make sure you up vote my answer! ;)

Answer 6

There is one more answer to this question and since I often have controversial opinions I'm ok with downvotes. That's the price for non-mainstream thinking...

It has usually nothing to do with how big or good your answer is. If it's a simple question then you'll get a lot of votes - even from answers that resemble those on SO and with just a couple of sentences or lines of code.

Most of the time it doesn't matter how good your answer is. If you are answering questions about some topic or technology that no body knows or understands you are not going to get too many upvotes - or maybe even none. This is how it works. People don't usually vote for questions (or answers) they don't understand and cannot verify their correctnes - especially if it goes way beyond naming conventions and good practices.

Some people are very strict about it and if they don't know what you are talking about you are not get a vote. I'm not, I sometimes blindly upvote answers if I have the impression it contains some useful information even if I cannot verify everthing myself.

The d3.js area certainly is one of these topics that very few people understand. There are barely 34 questions tagged with it.

You can make the answers more valueable and verifiable if e.g. you add some links to the docs or quote the docs.

---

And sometimes you just get votes because you already have a few of them... and other people think your answer must be good because others already voted for you. The same applies to downvotes. If your counter shows -1 then you are most likely to get another downvote then an upvote. It sometimes all depends on what comes first - was it a 1 then you'll be upvoted, was it a -1 you're most likely to be downvoted.

It usualy works this way with hot-questions. The most upvoted answer is the one that has most upvotes before the question went hot even if other answers might be better.