posted by Thom Holwerda on Mon 20th Oct 2008 15:58 UTC
IconHere at OSNews, we use a moderation system where your peers rate your comments, and where the OSNews staff rarely intervene. This system was put in place after realising that the editorial moderation was failing miserably; it had become too much work. Sifting through ten reported comments a day is one thing, sifting through 100 of them each day is another. The result was that editorial moderation had become willy-nilly, which led to understandable user frustration, and pointless email rants back and forth between users and staff members (mea culpa). The answer to this problem turned out to be a two-step process: banning anonymous commenting, and our peer moderation system. While the moderation system has its flaws, it has exceeded all of our expectations in making sure that our comments' sections are free of spam, and relatively fun to read. Still, we realise problems exist, but we also see a lot of misinformation floating around. The treat we have in store for you today will help in fixing some of that.

Because of our moderation system, we have access to a wealth of information regarding how our users moderate, what they moderate, and who they moderate. Obviously, we admins have access to that information via OSNews' backend, but you guys didn't get to see any of that information. A few days ago, we decided to change that, and create a page full of statistics regarding moderation, commenting, users, and stories. You can find it at osnews.com/statistics/.

Let's walk through some of these figures. It's good to see that there are three times as many "up" votes as there are "down" votes - meaning most comments are actually rated positive. This reflects upon the average comment score, which is 1.77, and only 13500 comments are rated below 0. Some people claim that down-modding affects the flow of comments in a negative fashion, but from the figures we can deduct that only 0.01% of the total amount of comments falls below the average view threshold, which means that there are very, very few 'broken' comment threads.

Often, the OSNews staff gets accused of heavy-handed moderation, heavily influencing the comment and trust scores of users. As you can see in the figures, this is absolute bogus. All admin moderation together accounts for only 0.79% of the total amount of moderation points given. Limiting the OSNews staff to David, Eugenia, Adam, and myself, it means that each of us only account for 0.197% of total moderation points given. Also interesting is to see how many comments we actually removed (designated as "invisible" - we can't actually remove comments) - 578. Of a total of 730000 comments. That's 0.08%. The conclusion we can draw from this is that the influence of the staff on the moderation system is not even remotely significant.

I also want to say a few words about the little tantrum OSNews user kaiwai placed in our conversations section. As it turns out, he was indeed systematically moderated down by two people - which sucks balls. However, the interesting thing here is that this blatant abuse actually showed how well the moderation and trust systems work: despite systematic downmodding, kaiwai's trust rate was the third highest of all our users - the abuse didn't affect him at all. Trust rates are calculated using all sorts of fancy algorithms (Adam knows all about those), and among other things, it shields users from this kind of abuse. The key here is to not be blinded by the scores of your individual comments - you have to look at the bigger picture.

Rests me to say that we are always tweaking our moderation system and the algorithms at work, and of course, there's still enough area to improve. Suggestions are always welcome, and if you would like to see a certain statistic added to the new statistics page, drop us a comment or email.

e p (3)    96 Comment(s)

Technology White Papers

See More