35
\$\begingroup\$

For a few months now I've been collecting specific data on the metrics of all SE sites throughout the network, and there's a disturbing fact that has arisen.

This data isn't yet public on my end, but it's almost there. In light of that I wanted to bring a certain concern to the communities attention.

When I was going through and reviewing some of our data, I noticed that our site (Code Review) is ranked very low on the voting rank. When we compare ourselves to only graduated sites, we rank in 56th on the Votes Per User scale. Meaning, there are 55 graduated sites that have a higher Votes Per User rate than us, considering there are only 71 graduated sites in the system, that puts us in the 21st percentile. We only have a higher Votes Per User rank than 21% of graduated SE sites.

What is the raw value? Well it's 4.54, that means that for all of our users, we have an average number of 4.54 votes per user. In the lifetime of CR, we have 4.54 votes per user.

The highest value of all graduated sites is Mi Yodeya, at 35.48 votes per user. This should not be a hard value for Code Review to achieve.

If we consider all non-Meta sites we are in 104th place, out of 159. That's only the 35th percentile. We're still lower than 65% of all non-Meta SE sites.

What is the breakdown? Of the 103 sites ahead of us, 1 is a Closed Beta, 55 are Graduated, and 47 are Open Beta. Almost half the sites we rank below are beta sites. We rank below 29 Meta sites.

Site State  | Total Above | Total Below | Highest VPU | Lowest VPU | Average VPU
------------+-------------+-------------+-------------+------------+------------
Open Beta   | 47          | 39          | 27.79       | 1.35       | 5.67
Meta        | 29          | 128         | 10.35       | 0.30       | 3.26
Normal      | 55          | 15          | 35.48       | 0.82       | 8.51
Closed Beta | 1           | 1           | 11.62       | 1.49       | 6.55

If we instead look at Votes Per Entity (where an entity is a question or answer) we end up with an even more disappointing chart:

Site State  | Total Above | Total Below | Highest VPE | Lowest VPE | Average VPE
------------+-------------+-------------+-------------+------------+------------
Open Beta   | 38          | 48          | 7.96        | 1.36       | 4.33
Meta        | 97          | 60          | 19.74       | 1.87       | 5.14
Normal      | 28          | 42          | 16.11       | 1.22       | 4.59
Closed Beta | 1           | 1           | 6.94        | 0.72       | 3.83

Our Votes Per Entity is 4.42, which puts us at 165th of 316. If only counting graduated sites, we're 29th out of 71.

Finally, if we count Votes Per Day as the metric, and consider the number of days since launch as the number of days since the first public appearance (the date of the first Closed Beta, if none then the date of the first Open Beta, if none then the date of the Launch), we rank 30th out of 71 with graduated site, but we rank 32nd out of 316 when compared to all sites (including Meta's and betas).

Site State  | Total Above | Total Below | Highest VPD | Lowest VPD | Average VPD
------------+-------------+-------------+-------------+------------+------------
Open Beta   | 38          | 48          | 103.45      | 2.09       | 16.20
Meta        | 97          | 60          | 108.77      | 0.19       | 3.52
Normal      | 28          | 42          | 20318.85    | 4.92       | 431.37
Closed Beta | 1           | 1           | 46.00       | 12.07      | 29.03

Note: all this data is as of 2 August 2016 at approximately 00:05UTC+00:00.

The gathering of this data is as follows: each night I have a program which crawls the /sites API endpoint on SE, and gathers the list of all sites in the network. Then, after that is complete, it crawls the /info API endpoint for each site and submits all data from that endpoint to my database. I then have a SQL query which I run on the database that reports all these metrics via multiple views and grouping.

I'm currently putting together the finishing touches on a site where all this data will be hosted, allowing anyone who desires to go in and run queries against it.

If there are any other metrics as far as Votes Per Day, etc. that anyone thinks would be nice to have in this meta post, comment with the metric and I'll try to gather it around. (Or find me in The 2nd Monitor.)


All this is coming to the following:

  1. Is the lack of voting on Code Review a problem?
  2. If so, what, if anything, can we do to improve the voting participation by users?
\$\endgroup\$
  • 8
    \$\begingroup\$ Could you clarify how you count users? Conventionally, Stack Exchange staff only consider users with ≥ 150 rep ("avid" users) when compiling these kinds of statistics. \$\endgroup\$ – 200_success Aug 2 '16 at 18:28
  • \$\begingroup\$ @200_success The number of users returned is determined by the SE API /info endpoint: api.stackexchange.com/docs/types/info and it appears to be the total number of registered users on the site. \$\endgroup\$ – 410_Gone Aug 2 '16 at 18:29
  • 2
    \$\begingroup\$ Comparing Code Review voters and Mi Yodeya voters shows that the situation is not necessarily as bad as your analysis makes it seem. We do have voters, but there is probably a long tail of drive-by users. \$\endgroup\$ – 200_success Aug 2 '16 at 18:35
  • 1
    \$\begingroup\$ I definitely think we need stats on "Votes per Users-who-are-actually-somewhat-regular-on-any-kind-of-basis". Otherwise, we're measuring a different problem. \$\endgroup\$ – Kaz Aug 2 '16 at 18:38
  • \$\begingroup\$ @Zak Unfortunately, SE provides no mechanism for gathering how many users are active easily from the API. It only provides me a TotalUsers value. \$\endgroup\$ – 410_Gone Aug 2 '16 at 18:40
  • \$\begingroup\$ @EBrown I guess this is why they set the bar at 150 rep. \$\endgroup\$ – Kaz Aug 2 '16 at 18:46
  • 5
    \$\begingroup\$ Whatever the stats and numbers are, your vote matters nonetheless. \$\endgroup\$ – Mathieu Guindon Aug 3 '16 at 17:36
  • 1
    \$\begingroup\$ This query has a column showing when the users were last seen: data.stackexchange.com/codereview/query/169776/one-vote-short \$\endgroup\$ – Hosch250 Aug 3 '16 at 22:52
  • 1
    \$\begingroup\$ In support of @Zak It might be interesting to use participation badge counts as a measure of whether someone is a frequent visitor or not. \$\endgroup\$ – pacmaninbw Aug 4 '16 at 11:01
  • \$\begingroup\$ @EBrown To gather the number of users above 200 rep, you could scrape the 'total reputation' table on the right hand side of this page. It shows that there are only about 3200 users on CR above 200 rep, as opposed to the ~95k registered users. \$\endgroup\$ – angussidney Aug 8 '16 at 11:39
  • 1
    \$\begingroup\$ @200_success Well that's unfortunate. I vote a lot on SE, and I only have > 150 rep on 14 of my 54 accounts. \$\endgroup\$ – Chris Cirefice Aug 11 '16 at 20:41
  • \$\begingroup\$ @ChrisCirefice then you're a drive-by voter. I am also a drive-by voter on 44 of my 63 accounts. Try doing some editing to get your rep up if you want, it also affects whether you can vote in elections on that site. \$\endgroup\$ – Raystafarian Aug 16 '16 at 11:23
9
\$\begingroup\$

I made my own query using SEDE.

It looks at votes cast per person (no matter when they were cast). If you know how long the site has been around, you could calculate average votes per day.

It's an easy and fast query actually:

select count(id), avg(upvotes), avg(downvotes), sum(upvotes), sum(downvotes) 
from users

I use the above query to get the data for all users. For "new" users, I add where reputation<150 and for "avid" users, I add where reputation>=150.

Results:

------------- + Count + Avg Up + Avg Down +   Up   + Down 
-- All   users| 94897 |    4   |    0     | 395650 | 35461
--"Avid" users|  6284 |   51   |    5     | 322880 | 32620
--"New"  users| 88613 |    0   |    0     |  72770 |  2841 

The average is really brought down by the new users, who obviously don't spend as much time on the site (and therefore vote less). There's not really a way to force them to vote (many of them will likely never visit again). Some of them might not yet have the privilege (although this is rare), or may have no interest in doing so.

Personally, I think this community is very open with votes, especially compared with sites like Stack Overflow (where I have posted hundreds of answers that sit at 0 score).

I also ran the query against every site. I'm not going to make a table for that, since it's a lot of data. My query (which uses my first query):

declare  db_c cursor for select [name] 
       from sys.databases 
       where database_id > 5 -- skip master, temp, model, msdb, Data.SE

declare @db_c_name sysname   -- holds name of db after fetch
declare @sql nvarchar(max) -- holds build up sql string

-- result table
create table #all_users (
    name nvarchar(40)
    , users int
    , avgUp int
    , avgDn int
    , totalUp int
    , totalDown int
);

open db_c
fetch next from db_c into @db_c_name
while(@@FETCH_STATUS = 0)
begin
    set @sql = N'use '+ QUOTENAME(@db_c_name) +';
    insert into #all_users 
    select
    '''+ 
       QUOTENAME(@db_c_name) +
    '''
    ,count(id), avg(upvotes), avg(downvotes), sum(upvotes), sum(downvotes) 
    from users
    ';
    exec (@sql)
   fetch next from db_c into @db_c_name
end;
close db_c;
deallocate db_c;

select * from #all_users 

(The same modifications need to be done to check for avid and new users.)


I went a bit further and decided to see what the average score (avg(score)) is for posts that aren't closed, only including things that were posted since 1/1/16 (where Creationdate >= ''2016-01-01'' and closeddate is null). Code Review averages a 2.

I have graphs!

Metas included:

Graph

  • 104 sites are less than CR
  • 70 other sites also average a 2
  • 135 are more than CR

(This may seem worrying, but the Metas are higher scoring on average.)

No Metas:

graph

  • 76 sites are less than CR
  • 34 sites also score a 2
  • 44 sites are more than CR

That puts us above average. In fact, we are ahead of the Trilogy (0) and also Mi Yodeya (1).

\$\endgroup\$
  • 1
    \$\begingroup\$ Side note: Why isn't my <!-- language: lang-sql --> working? \$\endgroup\$ – Laurel Aug 8 '16 at 17:34
  • \$\begingroup\$ I am assuming that this might be something that is site specific, where we may have to request the highlighting for our Meta site in order for it to work... I am going to write a post \$\endgroup\$ – Malachi Aug 10 '16 at 16:56
  • 2
    \$\begingroup\$ Meta code highlighting \$\endgroup\$ – Malachi Aug 10 '16 at 17:02

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .