Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

What's wrong with voting?

By amazing in Meta
Thu Dec 14, 2000 at 11:40:43 PM EST
Tags: Scoop (all tags)

This submission attempts to address what I see as the problems with moderation on Kuro5hin.

It seems clear to me that Kuro5hin's voting is, to put it bluntly, badly broken.

The problems are many:

(a) Uncertainty as to what the various scores mean. I don't think anyone votes according to the same criteria.

(b) 'Adjustment moderating' - users vote a post way down to 1 only for someone to decide, although they don't necessarily believe it should be at 1, it's easiest to get it where they want it to go by voting it up to 5.

(c) Voting according to prejudice. I have seen people moderate every post by a user to 1 for no good reason.

(d) Voting according to personal opinion. Frequently a post will be moderated down simply because the moderator disapproves of the opinion expressed. Again, I think this problem comes through uncerainty as to how one should moderate - should a post be moderated to a high score because it is well-written even if you find the views expressed in it to be offensive (or wrong at least)?

Voting on story submissions is similarly broken. I, for example, tend to vote things off if they are boring. The reverse is true to a greater extent - a poorly written story will always see the light of day provided it covers an interesting topic.

Furthermore, the question of whether something should go on to the front page is even worse. There appears to be no rationale for this decision at all, and so the headlines on the front page are entirely random.


(a) Perhaps we should have radio buttons with 1=poor 2=substandard 3=average 4=good post 5=excellent (These could be improved)

(b) (c) and (d)

1. Remove scores until the post has been moderated. This removes the problem of adjustment moderation.

2. Get rid of score = mean of votes. This is very silly. Some algorithm based on the mode (for those who don't know, this is the most common score, and therefore removes 'adjustment' attempts) would be best.

3. Restrict +5 and +1. These are overused.

4. Perhaps get rid of moderation altogether. Apart from removing trolls/spam, I can't see that moderation performs any useful function on Kuro5hin - although the moderation system is apparently a reaction to Slashdot's, in many ways Slashdot's works better - restricting to +3 and above on Slashdot acts as a filter, whereas I don't see what function the scores have on Kuro5hin at all - other reasons for moderation on Slashdot (ultimately page impressions through the encouraging of karma whoring and moderation) don't, I believe, exist on Kuro5hin.

Regarding story submissions I suggest the following:

A more complete voting system:

Interest rate 0-5

Quality of story rate 0-5

If either the interest or quality were high enough, the story would be posted; if enough people think it's very interesting, to the front page; if enough people think it's very poorly written, even if it's interesting then dump it.


Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure


Should we dump moderation of posts to articles?
o Yes 15%
o No 84%

Votes: 97
Results | Other Polls

Related Links
o Slashdot
o Kuro5hin
o Also by amazing

Display: Sort:
What's wrong with voting? | 48 comments (44 topical, 4 editorial, 0 hidden)
I disagree. (3.72 / 11) (#1)
by spaceghoti on Thu Dec 14, 2000 at 01:09:31 PM EST

I think the system works, even if it doesn't work the way you think it should. If you take moderation personally, you need to re-think your priorities.

That said, I voted this +1 Front Page. Not everybody votes by bias.

"Humor. It is a difficult concept. It is not logical." -Saavik, ST: Wrath of Khan

It works? (2.60 / 5) (#3)
by amazing on Thu Dec 14, 2000 at 01:25:07 PM EST

> I think the system works

What do you mean by that?

As I write this, someone has rated your post as 2. What has this achieved? What use was it? What criteria do you use to say that the system works or worked? You must have a reason - or do you just like clicking on forms?

> If you take moderation personally,

No, not at all. I couldn't care less: I don't know any of you, and therefore whether you like my opinions or not is of no importance to me, particularly as succeeding in moderation doesn't even bring me the so-called benefits of +1 bonuses and high 'karmas' that some people seem to revel in.

At least that seems to serve some purpose (although in my view useless) - the ultimately pointless achievement of a +1 bonus or whatever (although, on reflection, this is not necessarily worse than achieving a high score in Quake III or catching a fish - whatever turns you on is cool by me).

I can't see that moderating your post to 2.00 even served any transitory purpose.

[ Parent ]
the primary purpose of rating comments (3.83 / 6) (#5)
by Anonymous 242 on Thu Dec 14, 2000 at 01:33:50 PM EST

The primary purpose of rating comments is to give lazy people like me a gauge as to whether or not I should waste my time reading particular comments. Time, being a valuable and scarce resource, I find it worth my while to only read through threads that have a large number of highly ranked comments.

And generally speaking, rating works. Well thought out comments with a rationale behind their points generally (but not always) get ranked higher than comments that are simply the "I think so and so" variety. I'll admit that rating isn't perfect, but neither has it been abused in a widespread fashion as of yet.

[ Parent ]

the secondary purpose of rating comments... (3.00 / 7) (#6)
by leviathan on Thu Dec 14, 2000 at 01:39:34 PM EST

...is to give lazy people like me a gauge as to whether I should spend my time writing particular comments. If everyone thought they were not worth reading, I'd keep my opinions to myself.

I wish everyone was peaceful. Then I could take over the planet with a butter knife.
- Dogbert
[ Parent ]
I will answer your questions. (3.80 / 5) (#8)
by theR on Thu Dec 14, 2000 at 01:46:42 PM EST

This is how I see the answers to the questions you posed by amazing.

What has this (a rating of 2) achieved? What use was it? What criteria do you use to say that the system works or worked?

There are only two things rating spaceghoti's comment a 2 achieved. One was, it helped determine whether or not the comment was hidden to untrusted users. The second is that it helped determine whether or not spaceghoti is a trusted user. No more, no less. As far as I am concerned, that is all ratings need to do. I will decide for myself how worthwhile a comment is as long as it is rated above a one. I do not need the ratings to tell me which comments are good.

What do you mean by that? (In response to, "I think the system works.")

I believe the only reason for moderation is to try to limit the effects of spam and comments that are clearly not appropriate. Because definitions of what is appropriate and what is not are highly subjective, perfection is impossible. I don't know if spaceghoti agrees, but I think the system is effective and filtering most of the noise on K5. I don't believe the system was or should be designed to do anything more than that, at least relating to comment rating. Story voting and submission probably could be improved, but I have no problem with the comment rating system.

[ Parent ]
What is this trusted user thing??? (3.00 / 1) (#41)
by speek on Fri Dec 15, 2000 at 12:43:46 PM EST

I been hearing people talk about it, but I've looked around and haven't found anything that tells me what it is, or how to find out my status. What is this all about? What does it affect? What are the rules?

al queda is kicking themsleves for not knowing about the levees
[ Parent ]

FAQ, extended comment viewing/rating (none / 0) (#43)
by kmself on Fri Dec 15, 2000 at 04:02:18 PM EST

First, I'd suggest you look at the FAQ. Though the docs could use some updateing. There's also plenty of discussion of moderation and Mojo at K5, you're an adult, look for it.

The primary purpose of Mojo is to identify abusive users and to rank their posts below the visibility threshold -- moderation value < 1. Trusted users have the ability to see these comments, and to moderate at 0. For a comment to get a 0 moderation, it has to either have been previously unmoderated, or the 0 moderations have to cancel any positive moderation the comment previously had.

And, no, you can't see your current mojo rating, though you can get a pretty good idea of what it might be by reviewing modertions from the "Your Comments" page.

That's pretty much it, at the moment.

Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

In answer to your questions (4.66 / 6) (#17)
by spaceghoti on Thu Dec 14, 2000 at 04:27:25 PM EST

My post was moderated to a 2 (as of the time of this writing, a 3.2), and whatever magic determines my mojo took away my trusted user status. Fire and brimstone! Injustice and righteous indignation!

Not really. It means to me that my posting was short on content, which is fair. Had I gone through your article and picked it apart paragraph by paragraph (and possibly sentence by sentence) I probably would have been rated higher. While I agree that people aren't moderating according to the spirit of the system proposed by Inoshiro and Rusty, that doesn't necessarily invalidate it.

Some people vote according to bias. Some people vote according to agreement or disagreement. And some people post based on relevance. While in a perfect world the last criteria should have the most weight, this is hardly a perfect world. And while it's a damned good website, this isn't a perfect website. People are going to consider how much they agreed with the post when they rate it. I confess to doing it myself, in part because my disagreement with the person I'm responding to leads me to believe that their arguments or logic is inherently flawed. If I stop to think about why I'm disagreeing, sometimes I'll realize that we're just approaching the issue from opposite sides. I try to only moderate down when I believe someone is being irrational or intentionally insulting. Otherwise I rate people based on relevance and content, which is how I understand the spirit of moderation to work.

What overall purpose does moderation serve? It gives us all a voice to tell you what we think of what you said without trying to get personal. I've got an editorial comment correcting a mispelling voted down to 1.66, which is absolutely ridiculous. If I wanted to, I could get pretty peeved with this. But it just isn't worth it. K5 readers are telling me I didn't say anything helpful or relevant to the topic at hand, and that's fine. I'll take the hits on my mojo because it really doesn't matter if I can view hidden comments or moderate a troll to 0 instead of 1. My life will go on, and I'll continue to enjoy the discussion here, while continuing to add my moderation to let people know what I think of their posts (or what I think of someone being modded down who doesn't deserve it).

So that is why I disagree. I think the moderation/voting system here is just fine. Maybe there are some improvements that could be made, but I don't think they're critical.

"Humor. It is a difficult concept. It is not logical." -Saavik, ST: Wrath of Khan

[ Parent ]
Comment ratings badly broken (3.50 / 12) (#7)
by maynard on Thu Dec 14, 2000 at 01:45:16 PM EST

Editorial: I'd argue that you should have Rusty or Inoshiro edit the submission to change the word "voting" to "rating" where you refer to posts and not story submissions.

I'm using "Topical" because I'd like to address the issue on comment ratings. I've given up on them. I've set my page to stop offering comment rating drop boxes and set comment sorting to unrated. Ratings are now worthless, they show the K5 community how well one is liked, or how many secondary accounts your debating opponent has to force his opinion up at a higher rating than yours. These are troll tactics, and they are widespread in use throughout K5 now.

Ratings were once simply an attempt to filter out those who posted meaningless drivel, but have turned into a political tool to attack members with whom others disagree. Some simply rate every comment a 1, no matter what that user writes. Others rate themselves up using secondary accounts. And a few honest members even let the rating system alone as a show of honesty. This is just like a prisoner's dilemma paradox, where the only way for the rating system to function properly is for the members to behave responsibly. However, each member can easily manipulate the rating system to his/her advantage, thus diluting it's value to the community as a whole. We are now at the point where so many manipulate the rating system, ratings are meaningless and instead of contributing value to the community they've turned into a weapon for tit for tat mojo wars between individual members.

There is only one solution. Remove yourself from the ratings game and forget mojo as a scale of anything. Rusty's biggest mistake with ratings was to tie a presumed value of arbitrary comments to a user's "mojo" contribution to the K5 community; thus giving select members who successfully manipulate the rating system special status within the K5 community. I argue that he should revoke those privileges until the K5 editorial staff work out another system for managing mojo, and either separate out mojo from ratings, or shut down the rating system altogether. JMO.


Read The Proxies, a short crime thriller.

Mojo just isn't that important. (4.33 / 6) (#14)
by ramses0 on Thu Dec 14, 2000 at 03:31:05 PM EST

If you somehow manage to acquire vast amounts of mojo (ummm, actually 5 is as high as you can ever go), then you get to look at crappy posts like this one:

Good to hear. (0.18) (#32)

Why don't you die about it?

It's not that great.

[ rate all comments , for great justice | sell.com ]
[ Parent ]

Might add -- half dozen posts in past week (4.33 / 3) (#30)
by kmself on Thu Dec 14, 2000 at 09:51:09 PM EST

Not only is the zero-vote a small priviledge, but it's a pretty small fraction of posts. A half dozen or so in the past week.

Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

OK, an eyeball war. (4.00 / 2) (#32)
by maynard on Fri Dec 15, 2000 at 12:58:29 AM EST

Right, the term "mojo war" is maybe off base. I've never had "trusted status" so I can't speak for what that's like, but then I'm really not pursuing it either.

However, when an individual user corrupts the rating system by misusing multiple accounts to force another user's rating down, or to force his/her rating up, what they're really doing is enacting in an eyeball war to controll what the majority of visitors bother to read. Most people don't bother to write, or even rate comments; only select few write regularly. I'd argue that it's mostly writers who bother to rate comments, because writers have a vested interest in promoting a particular viewpoint and sometimes even their own material. So, when we're looking at a person who abuses K5 by moving from one account to the next in order to stack a rating, what we're really looking at is an individual with a vested interest to protect, and either eyeballs he/she desires to attract or discourage.

A somewhat machiavellian view.


Read The Proxies, a short crime thriller.
[ Parent ]

Wow. The rating on this comment is really fucked. (1.66 / 3) (#18)
by 6502 on Thu Dec 14, 2000 at 04:58:50 PM EST

I mean, can you give a better example of an unfair rating? 0x55

[ Parent ]
Irony, or unclear on the concept? (4.00 / 1) (#29)
by kmself on Thu Dec 14, 2000 at 09:48:16 PM EST

Without specifying what the actual (at the time of your post) and reasonable values for this post are, your comment lacks any meaning. Unless that's your point. So is this irony or lack of understanding.

Moderation is in constant flux. It tends to converge toward a value, and the convergence is stronger over time (with more moderation). It's not perfect, but it's pretty good.

Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Bell curves (3.50 / 10) (#10)
by ucblockhead on Thu Dec 14, 2000 at 02:19:44 PM EST

3. Restrict +5 and +1. These are overused

What if there were some sort of bell curve enforcement. In other words, you only get one 4 vote for every four 3 votes. You only get one 2 vote for every 4 vote. You only get one 5 vote for every four 4 votes. You only get one 1 vote for every 5 vote.

That'd encourage more "average" voting and discourage those who want to come in knocking down everyone to one. It'd also make a 5 mean more.

Another thing that might help would be to say "3.00" on posts instead of "none" when no one's rated it yet.

I really don't see Slashdot's as superior at all. One big problem I have with Slashdot is that a lot of people have figured out that the quickest way to a "5" is to post something funny. In the last few weeks, it seems that the first few posts, when browsing at +3 or something, are not insightful opinions or informative posts, but some obvious, fairly lame joke.

This is k5. We're all tools - duxup

life is not statistics class (4.00 / 2) (#11)
by vsync on Thu Dec 14, 2000 at 02:32:30 PM EST

What if you run across a story in which every comment is exceptionally good? This has happened. I've also run into long runs of incredibly stupid posts. Let's not pretend we can enforce mediocrity.

"The problem I had with the story, before I even finished reading, was the copious attribution of thoughts and ideas to vsync. What made it worse was the ones attributed to him were the only ones that made any sense whatsoever."
[ Parent ]
A grade budget is a good idea (4.00 / 2) (#21)
by saucepan on Thu Dec 14, 2000 at 05:44:24 PM EST

A grade budget doesn't necessarily have to be balanced out after every story -- there are other enforcement policies that allow some slack.

It's true that, regardless of the mechanism, there would occasionally be a situation where you'd like to assign more high grades then you are able to: otherwise, the enforcement mechanism would be having no effect at all. This sort of sucks, but it's the old trade-off of short-term inconvenience versus long-term gain. With the proper setup it would be very unlikely that everyone would run out of 5's at the same time.

This old Slate article discusses the incentive problem in more detail in the context of college grade inflation.

[ Parent ]

Unrated (none / 0) (#37)
by kmself on Fri Dec 15, 2000 at 02:04:11 AM EST

Another thing that might help would be to say "3.00" on posts instead of "none" when no one's rated it yet.

Quite intentionally not. An unrated comment is essentially a NULL. It's not a 0, it's not a 3, it's not a 5, it's not 3.29. It's unrated.

Slashdot suffers this bug -- unrated comments are '1'. You can't tell if a comment is a '1' because it hasn't been rated or because it's been rated up and down. This is a very bad bug of Slashdot.

Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Moderation... (3.62 / 8) (#12)
by pb on Thu Dec 14, 2000 at 03:10:28 PM EST

I like how the moderation system currently is a lot better than I like your suggestions.

While you're at it, restrict '3'; that's *far* overused. I'd much rather see a good post initially modded to the top, and then settling down to its rightful place than have it start at 3 and be consigned to mediocrity. That's also why we have the option of putting the posts with no rating at the top...

But customizing the page viewing more works as well... I for one would like an actual "Flat, Newest First" mode; we all have our pet peeves. :)
"See what the drooling, ravening, flesh-eating hordes^W^W^W^WKuro5hin.org readers have to say."
-- pwhysall
Slashdot does it better (2.75 / 8) (#15)
by GreenCrackBaby on Thu Dec 14, 2000 at 03:39:44 PM EST

Before anyone jumps on me, let me clarify.

The Karma point crap is stupid, but I do like the way "the cream rises to the top." (their words, not mine) I like to be able to scroll through all the comments and have the good ones standing out. With k5, the rating system really does little to make the good stuff jump out at you.

The scale of 1 to 5 is useless as well. Just have a +1 or -1 like the comment moderation.

you think the cream rises... (3.57 / 7) (#20)
by cetan on Thu Dec 14, 2000 at 05:14:07 PM EST

unfortunatly, the cream does not always rise to the top on /. (for example, to pick a name, at random, say Signal11 ? [now watch this post get rated to 1...]) As CyberQuog points out above, only the "group think" posts rise to the top. Whatever flavor of the month is popular is moderated up, regarless of whether or not it has value. People think "well, the poster used big words and added something about GNU so it must be an intelligent post." and mod it up. Forget that the post is utter crap, it's a +4 Informative!

Sarcasm aside, I have noticed some problems with K5, like all of a sudden all comments being rated at "1" for a particular story. Obviously this is a sign of abuse. How to stop this? I don't honestly know...

===== cetan www.cetan.com =====
[ Parent ]
/.: false negatives in moderation (4.33 / 3) (#23)
by kmself on Thu Dec 14, 2000 at 07:01:39 PM EST

This is one of two major concerns at Slashdot. While the false-positive count is low -- most of the stuff that gets moderated up is indeed good -- the false-negative rate is very high -- there's an awful lot of good stuff which doesn't get moderated up. Especially well into a discussion, it's hard to identify recently added posts worthy of moderation. This accellerates the end of a discussion as there's little incentive or possibility to continue conversation.

Worse, the system has a very strong positive feedback cycle, leading IMO to the karma whoring phenonenon. Karma provides an initial point boost, these posts are further moderated up, karma increases. Likewise, negative moderation keeps posts where they can't be heard. While this is desirable for true trolls, it's annoying and aggrevating to the rest of the crowd. The kluge currently is to cap both moderation (to 5) and karma (to 50).

Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

You're right (none / 0) (#39)
by GreenCrackBaby on Fri Dec 15, 2000 at 10:22:49 AM EST

The cream doesn't always rise to the top at Slashdot. However, it does rise. That was my point. Here on K5 there's nothing (aside from a number on a post) that tells you "Hey, others found this post interesting/funny/etc."

[ Parent ]
Don't use it (3.50 / 8) (#16)
by Eimi on Thu Dec 14, 2000 at 03:42:41 PM EST

If you feel the ratings are useless, then don't use them. That's always an option. Beyond that...
a) Uncertainty as to what the various scores mean. I don't think anyone votes according to the same criteria.
I've heard a lot of people complain about this, but I really don't understand what you mean. What's so hard about 1=worthless, 5=fantastic, and everything else in between?
(b) 'Adjustment moderating' - users vote a post way down to 1 only for someone to decide, although they don't necessarily believe it should be at 1, it's easiest to get it where they want it to go by voting it up to 5.
What's wrong with it? Have you seen comments that have a score you think they shouldn't, and somehow blame that for it? I guess I don't see the problem. People rate comments however they want, and if they feel the need to overreact to correct something they strongly disagree with, let them.
(c) Voting according to prejudice. I have seen people moderate every post by a user to 1 for no good reason.
Yeah, I've seen that too. It's unfortunate that it happens, it scuks, but you deal with it. Hopefully a single 1 won't be too much of a hit if a large number of ppl feel the comment is worthwhile. You just made a pretty good argument for "adjustment moderation", actually. If you see something that's been pointlessly moded down, mod it up!
(d) Voting according to personal opinion.
Again, unfortunate, but I don't know what can be done about it. Hopefully the majority of K5 viewers will be able to distinguish between a poorly presented or poorly thought out comment and one that they disagree with.

I really don't see what's to be gained by abolishing moderation. If you feel it doesn't work for you, then ignore it.

Moderation (3.50 / 8) (#19)
by CyberQuog on Thu Dec 14, 2000 at 05:07:12 PM EST

>in many ways Slashdot's works better
I find Slashdot's moderation to be horrible, usually only the "group think" opinions are moderated up. Alot of which are ignorant, troll's and the like. I like Kuro5hin's moderation because I find that it promotes discussion instead of intimidating people. How many times have you posted to /. and no one even read your post.

Also, I don't think the K5's moderation is useless, I use it to sort the highest rated comments to the top. I do think the rating's are sometimes confusing though, and should be labelled somehow. If you care (I doubt anyone does) I have my settings; View: Mixed, Display: Nested, Sort: Highest First, Oldest First, and Rate: Yes.

An idea (2.00 / 2) (#22)
by k5er on Thu Dec 14, 2000 at 06:17:20 PM EST

How about simply adding names to the voting scheme. 1 means you totally disagree with the comment, and 5 means you totally agree with it. So if the article has a low average, it means most people tend to disagree with it and its probably a badly written or uninformed comment.

However, it doesn't matter how it's done, some people will be happy, and others will bitch about it.
Long live k5, down with CNN.
Aren't ratings a measure of contribution? (2.66 / 3) (#27)
by maynard on Thu Dec 14, 2000 at 08:26:30 PM EST

Really. Why should one rate a comment based on how well they agree with it's contents? If everyone does that then the only comments that will get rated up will be those with which the majority agrees, and isn't that exactly what folks complain about Slashdot moderation? Don't you think it makes better sense to rate a comment based on how well it promotes healthy discussion, how well it's written, and how much of a contribution that comment makes to a particular story? Because otherwise, we've simply got a popularity contest on our hands. Do we want K5 to be a club or a means for discussion? That's the issue.

What we have here is a prisoner's dilemma paradox whereby the only way the system can work is if every user behaves responsibly, but any user can defect and gain advantage by manipulating ratings through multiple accounts and collusion with others. Given the rules of this game the only way out is not to play. However, scoop could be encoded to show transparency to the ratings process, so that it would be impossible to rate another without everyone knowing. This won't prevent multiple account holders from manipulating the system, but it will hold those who use a single account to a consistent standard of conduct.


Read The Proxies, a short crime thriller.
[ Parent ]

Not the purpose (4.00 / 1) (#36)
by Elendale on Fri Dec 15, 2000 at 01:45:46 AM EST

That isn't the point of rating comments anymore (never really was in k5). The point to comment ratings is to rate up users who are of value to k5 and rate down users who would likely abuse k5. Really. The only reason to have mojo is to give users the ability to delete (in the form of moderating to 0) comments- and users have to be really trusted to be given that power. While good writing is generally an indication of commitment to the site (you need to be committed to write good comments) its not always the case.


When free speech is outlawed, only criminals will complain.

[ Parent ]
Agreement with a twist (3.80 / 5) (#24)
by Freedom_2 on Thu Dec 14, 2000 at 07:07:09 PM EST

I agree with the intent of this story that voting/rating is not working the way 90% of us would like it to work.

Don't get me wrong, it actually works in the usability sense, but fails in the fact that each person is ranking and voting based upon very different standards (i.e. their own subjective reasons).

I have thought on this subject a couple of times also and IMO there should be a variety of rating/voting choices. This gives us two major pluses:

  1. More accurate reasons for the rankings to help writers make better decisions about their next story
  2. More accurate ways to sort/filter/rank stories and comments by the readers

So someone with a pet peeve against spelling errors or poor writing could filter out or down the comments rated as such, while someone you could care less about the writing could still get to the comment based upon it's higher rating in content, etc.

I think there should be a variety of ways to rate (And knowing nothing about Scoop?, I do not know if it is feasible or not). Here are a few examples I think would deserve consideration: (I think amazing's examples may be too broad)

  • Grammar and Spelling
  • Posting Party (so you can rate that troll you really hate)
  • Interest in Topic (I think this is what the current rating system is intended to be for)
  • Uniqueness of Topic (some topics are covered way too much, and some too little)
  • Currentness of Topic (a topic might be old news and/or old hat)
  • Validity of Comments (because you might like the topic but think the comments on the topic are worthless, etc)
  • Offensiveness

Maybe too much, maybe too little. But giving posters feedback on their posts, and users ways to filter based on reasons why they rate in the first place would be a good thing.


Why voting doesn't work. (3.20 / 5) (#25)
by enterfornone on Thu Dec 14, 2000 at 07:08:52 PM EST

See the link in my sig for a better explaination. In summary, I'm not complaining because my posts get voted down (we can't see mojo here, but I doubt mine is much more than 2). I'm complaining about the fact that voting discourages discussion. People would rather vote than post, and others are scared of posting things that are likely to affect their precious mojo.

Voting doesn't actually achieve anything. Unlike Slashdot where moderation serves as a way of avoiding spam and trolls, on K5 spams and trolls are either deleted or in the 0-1 range. The 1-5 range does not serve any useful purpose.

efn 26/m/syd
Will sponsor new accounts for porn.
Why moderation is bad (3.16 / 6) (#26)
by Sunir on Thu Dec 14, 2000 at 08:16:48 PM EST

Some of you may have already read my opinions on moderation, so given that and I'm sleepy, I'll keep this short.

Mojo will always result in negative behaviour because Mojo is entirely negative. That is, it is punitive. The best you can do to encourage good work is not punish someone as much as you would otherwise.

Consider, if the goal is to push people towards higher moderation values in order to put them at the top of the list, then that is the natural state of the material. Every vote other than 5 is a vote down.

Since negativity fosters negativity, certainly there will be issues with Mojo. On the other hand, my other stated opinion is that Mojo is really elegant. It is quite stable against attack. Then again, maybe a reaction to violence is not the best way to form a social system.

Slashdot moderation is equally punitive by being a reinforcing cycle. The punishment there is ignorance; denying people their voice. However, it does actually encourage people with the +1 ratings, which are cumulative. In that case, however, it suffers from the folly of rewarding A while expecting B. It rewards mass appeal in the hopes of achieving quality. However, as any stable democracy knows (ironically), mass appeal is the last thing you want.

Ultimately, placing a numeric objective scalar value on a subjective, ill-defined vector of qualities is going to shoot you in the foot. While it is true (as Karsten?? points out elsewhere) that people can loosely scalarize their "feel" for a post, that ability gets worn out with practice. The more you think about, the harder it becomes. Try moderating 100 posts in a hour.

A better way, I think, would be to use English-for example the editorial comments--to help the original author improve the article copy by making suggestions. As for individual comments, that's tougher. But is there more value in 200 redundant comments or 5 well written pages? Whatever. Collaborate, don't moderate.

"Look! You're free! Go, and be free!" and everyone hated it for that. --r

Irony: I stop rating yet see wonderful comment (4.00 / 2) (#28)
by maynard on Thu Dec 14, 2000 at 09:04:55 PM EST

This is a truly insightful comment. It's impossible to promote the creation of new content (or anything) through punishment. How well would a whip master cajole a herd of writers into producing literature? Not that comment rating is the same as being whipped, it's just an analogy. I wrote about this being a prisoner's dilemma paradox, I'd love to see your response to that point.

I really think the system is badly broken by allowing the corruption of collusion and multiple accounts, while never promoting playing by the rules. One user with one account will always be beaten by this system, so individual users with iconoclastic or unrepresented views are simply pushed away, without consideration given to their contribution or involvement. Slashdot avoids this by tightening the reigns around moderation through meta-moderation, with the result that a different view, as long as it's reasonably well written and not offensive, will usually get through meta-moderation. K5 has no secondary mechanism to handle those who misuse ratings to push their own personal gripes, agenda, or who collude with others. That's a serious problem.

I think just about every regular writer has been bitten by this, so it's now reaching a tipping point where the general user community is getting annoyed by how comment ratings are being mis-used. That's why there are so many repeat submissions on this issue... each one is a personal gripe by a user who mustered up enough courage to buck the system and expose it's flaws. And don't think that this goes without risk... Rusty likes to say that there is "no K5 cabal", but there most certainly is; and they have a vindictive willingness to punish with glee those they do not like. In the end, will such behavior promote or discourage intelligent and creative commentary? Do we want a K5 club, or a discussion forum which represents a diverse set of views? That's the issue.


Read The Proxies, a short crime thriller.
[ Parent ]

Content, quality, conversation, wiki, incentive (4.50 / 2) (#38)
by kmself on Fri Dec 15, 2000 at 07:08:08 AM EST

Hello again, Sunir.

I disagree strongly with several of your points, though you do raise some good ones.

Starting at the bottom, there's a distinction between what a site like K5 or Slashdot is trying to accomplish, and what a Wiki or collaborative editing tool does. Weblogs are conversations, the focus is on threads, discussion, and preserving a full record of the conversation, while at the same time illuminating the highlights and winnowing away the chaff. Collaborative editing tools are about creating a (usually) interlinked and interconnected set of documents. Very different goals, very different mechanisms. Both extremely interesting, and both very much on my mind.

With that statement of purpose, a weblog needs some means for identifying and cultivating useful content. For a small enough board, eyeball evaluation is sufficient -- the volume isn't so large that a user can't identify useful content herself. With size, other methods are required. If what is being measured is some likelihood or probability of interest, then a scalar value is precisely what you're looking for: any one user is going to have some ideal likelihood of interest in reading a particular post. The question being answered is: on a scale of 0 to 5, how likely am I to be interested in reading this post?

I'm not saying that Scoop moderation, of itself, accomplishes this goal. I am saying that:

  • Moderation is better than no moderation -- and if it isn't the reader is welcome to ignore moderation, excepting hidden comments.
  • Scoop's moderation system has several key advantages, spelled out at length in another post, over the method implemented at Slashdot. I agree with your assessment of Slashdot's reinforcing cycle.

Scoop's moderation score is one of several metrics which may provide a proxy for the ideal interest value a reader may have for a particular post. It's a general measure. As currently implemented, it's a global measure -- all readers see the same score on a comment. However both these characteristics could (theoretically) be modified. Keyword or context scoring might identify posts with a greater or lesser interest for a reader. Selective inclusion of moderator assessments, either manually selected or driven by software, could identify posts a reader might want to read based on the assessments of a specific set of raters. The system is flexible and extensible.

As applied to the submission queue and the editorial process, your post has a great deal more applicability. I've been frustrated by the submission queue for much of the past six months, and it's not getting better. Complex measures are necessary, and a simple numeric score is not enough. However, that's not the core focus of the conversation here, and you're confounding the issue.

I disagree rather strongly with your assessment that moderation (and mojo -- though again, you're confounding the issue) are strictly negative. They're not. Content starts its life on K5 as neutral, as posted by virtually all users (untrusted being the only exception). Moderation adds value to the content by indicating its relative worth, which may be lower, or higher. If your problem is one of assigning relative value and/or merits to things, I'm afraid we have a fundamental disagreement. If your issue is that any particular measurement is going to have a degree of arbitrariness and variable appropriateness according to different individual preferences, we're in agreement. But I return to my previous point: Moderation is better than no moderation, in a sufficiently data-rich environment.

Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Change of venue (4.50 / 2) (#44)
by Sunir on Sat Dec 16, 2000 at 02:32:45 AM EST

I was hoping you'd disagree. You always have something insightful to say, and I'm certain I didn't get it right. I have a lot to say back, but... Given that:

  • As you know, my goal here is not limited to k5, but to finding a decent resolution to the given forces so that k5 and others can do the right thing;
  • As Mojo is very good, I want to describe its forces and its resolutions so we can move the art of story moderation forward;
  • I'm getting tired of making the same points, as well as forgetting them;
  • Others seem to be making the same comments over and over again as well; and
  • For purely selfish reasons, I'm getting sick of writing in pseudo-HTML, especially linking back to other concepts I've written about before
I move for a change of venue to MeatballWiki. In this case (at least!), I think the wiki format would be superior to a web log. I'm going to reply there if it's all the same to you.

I would appreciate it if you copied (or allowed me to copy) your reply there. Same goes to anyone else interested.

"Look! You're free! Go, and be free!" and everyone hated it for that. --r
[ Parent ]

You'll be glad to hear... (4.00 / 2) (#46)
by kmself on Sat Dec 16, 2000 at 03:15:02 AM EST

...there's a good chance of a Wiki showing up at K5. Easier than Rusty writing and administering docs himself. We'll see how it scales to a potentially large population....

Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

False Mojo, comment rating equivocation (5.00 / 1) (#47)
by Sunir on Sun Dec 17, 2000 at 03:09:38 PM EST

Once again, I'm wrongly using the word "Mojo" instead of "comment ratings." I think the comment rating system is entirely punitive. Mojo is not entirely punitive, unless you count being a trusted user is punishment as you get to read all the spam.

Thus, please reconsider what I've said given this error.

"Look! You're free! Go, and be free!" and everyone hated it for that. --r
[ Parent ]

Wow, another one.. (3.71 / 7) (#31)
by Inoshiro on Thu Dec 14, 2000 at 10:39:29 PM EST

People moderating based on personal biases, and otherwise doing things people don't like? No way, dude!

[ イノシロ ]
K5/Scoop Moderation Backgrounder (4.61 / 13) (#33)
by kmself on Fri Dec 15, 2000 at 01:20:16 AM EST

A word of introduction -- I'm one of the people who's had a lot of influence in designing the Mojo and moderation systems at K5 and in Scoop. When I first stumbled over the site a year ago, I realized it had the potential to avoid a lot of the mistakes which Slashdot had stumbled into -- rather nice of them to do so, as they showed us what minefields to avoid in the process.

The following is abstracted from an email exchange between myself, Rusty, and a K5 participant, and a few other random documents. Though I'm not distinguishing among them, my foil in this discussion is actually a composite personality.

I believe this should help address some of the design intentions, pitfalls to be avoided, concessions, and limitations, to the Scoop moderation engine.

I'll also add that my own comments are rated over a wide range, some of my stories get posted, some don't, and I've got my own gripes about the system. But I think it's fundamentally right.

Fairness, Group Think, Filtering Tools

The issue of "fairness" is such a sticky one because it's such a personal assessment.


The biggest problem I see with ratings is that people really are using them to push ideological views and personal biases at the expense of those their marginal views.

I think it's deadly, I think group think is a cancer.

Think of this as a feature, not a bug. First, people will do this, it's human nature. The goal then is to come up with a system which identifies posts in which agenda-pushing (or just plain radical disagreement between two or more camps) exits. IMO this will be apparent in the statistics behind the post -- standard deviation, a measure of variance, will be large for these posts. Computing and displaying this need to be implemented, but the problem can be identified.

K5 moderation is an amalgam of many things. Both personal opinion, and personal response to differing opinion, are going to be part of it. I will moderate down posts I feel are just plain dumb, while I'll give credit to a thoughtful response to views similar to mine. Depends on context, mood, etc.

Similarly, K5 users cannot be authenticated -- there's no way of preventing me from creating multiple accounts, or sharing a single account with multiple people. While there are systems which deal with the issue of strong authentication (e.g.: online voting schemes), the authentication step is assumed.

There is no technical fix which is going to change these facts, they are a features of the system. Collaborative filtering and positive incentive mechanisms must take them into account as givens.

There's a distinction between group-think (everyone voting the same) and a difference in opinion resulting in a strong degree of disagreement with a comment's true value. The latter, as I said earlier, will be apparent with additional statistics such as standard deviation as a conflict measure. This was part of my original suggestion for the Scoop moderation scheme, and continues to be a lapse of the system. Specifically, I'd like to see the following statistics. Note particularly how I term them -- these are proxies for some real quantity, however, they can only approximate (and sometimes poorly) that measure:

  • n: Number of moderations. Proxy for interest.
  • mean: Average value of moderations. Proxy for "value". This is what moderation currently provides.
  • std dev: Standard deviation of moderations. Proxy for controversiality.

...adding the ability to filter according to complex rules (show me only highly rated or highly controversial stories with more than 5 ratings, and all stories with fewer than 5 moderations), will product a system which can both support high S/N and be relatively free from abuse of various sorts.

Another means to the same goal is to allow inclusion or exclusion of specified users opinions in your rating scheme -- essentially coming up with a "buddy list" of editors. This could be a manual or automated process, or a bit of both.

Yet another idea I like immensely is the ability to apply a personal bonus/decrement value to a member's score. This is more flexible than blacklisting (essentially slapping a -5 on a user), and could itself be used to provide feedback to the system.

The problem with conferring trusted status upon people who write good posts is that it does not follow that a person who writes well is trustable (demagogues; did you read Ender's Game? Locke/Peter) and it does not follow that a person who does not write well is untrustable.

This is called in managerial contexts as "The folly of rewarding one behaviour to get another."

Very true. It's a complex problem of itself, and I've been thinking of ways in which it might be improved. Essentially, you've got three behaviors at K5:

  • Writing (both stories and comments)
  • Moderating (again, stories and comments)
  • Reading

Moderation works reasonably well. There needs to be (IMO) more of it, but of generally quality moderations. What's a quality moderation? Good question. Some ideas:

  • One which helps establish the status of a post. Early moderations or moderations or moderation to posts which have few or no moderations count a lot.
  • Moderation patterns which tend to agree, within reason, with the group consensus. Someone who's consistently an outlier probably isn't feeding much signal to the system.
  • Moderation patterns which don't slavishly follow group trends. Moderation works best by increasing differentiation between comments (preferably in a meaningful way). Note that a moderate devation from the norm is quite healthy.
  • Patterns which aren't pedantically consistent. The guy who mods all 1s or all 5s isn't adding much to the system.
  • Clustering -- moderators who cluster very strongly together, particularly in an absence or negative sense to outsiders, may be trying to game the system, and should at least get some administrator scrutiny.

On Goals

What's the goal? Maximize content quality. This means encouraging both good writers and good moderators. Your population is composed of those who write (who may also moderate), those who only moderate (but don't write), and those who read. We can ignore the problem of trusting Readers, simply because they don't participate in the system. I would, however, like to be able to both assess moderation quality, and find a way to reward those who moderate well, though they may not write.

The current Mojo scheme is a bit of a hack, but it seems to produce mostly good results. The real key is to identify the outlier Really Bad Elements that come along, and quickly. Trust and trusted member priviledges/rights are a possibility, but really a secondary goal.

Moderation Frequency, Reputation Attacks

Since not enough readers rate comments a few poor ratings can destroy an honest contributor's "Trusted" status.

This is a bug in the system. Moderation by small numbers of people should have less influence than moderation by large numbers of people, all else being equal. The Mojo calculation algorithm needs to be modified to accommodate this. Rusty and I are still discussing why and how.

Encouraging moderation is another issue. Ultimately this feedback should also be built into the system so that people have an incentive to contribute quality moderation.

Karma Whoring

This is exactly the same problem with "karma whoring" on Slashdot, those who know how to emotionally motivate moderators get the reward.

Crucial distinction, frequently iterated.

  • Slashdot karma is cumulative, confers special powers, and tends to be self-reinforcing. The fact that both karma and moderation at Slashdot had to be artificially constrained (1-50 and 1-5, respectively), should be a clear indicator that something is wrong.

  • Scoop moderation has several crucial properties:

    • It is bounded, restricted to a range of 1-5.
    • It is convergent -- with more moderation, a single final value emerges from the noise.
    • It is independent of the number of people moderating -- a comment's moderated level can range from 1 to 5, regardless of whether one person has moderated or 100.
    • It is continuous. By allowing fractional values (in reality, two decimal points), Scoop allows for many distinct moderation values, making arbitrarily fine distinctions in ranking possible (though not necessarily meaningful).

    Mojo shares many of these attributes, but modifies the convergence rule: it is short-term convergent (in the short term, more moderations tend to converge toward a value), but weighted toward recent activity. It also does *not* convey special privileges in posting comments (e.g.: boosted score).

Slashdot karma allows "whores" to accumulate a quantity which is not immediately responsive to current behavior, which tends to reinforce the score (more highly moderated posts tend to get more moderation, moderation score is not independent of number of moderators, quite contrary), and abusive behavior becomes possible. While it's possible to get an elevated K5 status, it takes continued effort to keep it -- and the powers conferred by it are rather thin.

I've explained bits of this at the Scoop website somewhere, in a followup comment to an article on moderation. (Unfortunately, this comment did not survive the relaunch of Scoop to a new site -- the comments database was corrupted).

Slashdot False Negatives (Undermoderation)

Because of this I've given up on reading Slashdot with a threshold set and read it uncut at -1; there are simply too many gems which sit at 0 or 1;

See above discussion of K5 moderation. Malda claims Slash moderation is effective because setting a cutoff of +2 or +3 screens out bad posts. And yes, the false positive incidence is low. However, the false negative incidence (high quality posts with low moderation) is very high. Moreover, the likelihood of a post, however worthy, of being moderated up to a viewable level falls drastically as a discussion ages, discouraging additional contributions to the topic.

Moderation Quality -- Factual/Technical Assessments

often technically accurate posts which never gain moderation approval. And this is the part where I vigorously disagree, Rusty. Those statements which can be verified as accurate and factual, technically or by reference, should always gain a higher rating than those which simply promote a personal opinion. If one can defend their unpopular views with a strong argument backed by references, as in academic debate, IMO this should always have an advantage over simple personal opinion.

Agreement in goals, disagreement in degree. I'd say that the factually supported arguments should tend to have higher ratings. I'll allow for occasional lapses and variance. Imperfect worlds, and all that. Unfortunately, a rigid implementation of this sort of rating requires some sort of recognition of a moderator's knowledge and authority in a field, and must recognize that a strong background in one area doesn't necessarily translate to others.

Moderator Bias

About bias vs. objectivity in rating: I think the main problem here is that comment rating serves two somewhat unrelated purposes. We each see one of them as more important which colors our view of how rating should be done. Here are the two basic things rating does:

In one sense, rating comments simply provides an ordering mechanism


That is not to say that the dissenting view should be suppressed, especially a good expression of it, merely that one facet of rating can and, I think should, highlight the view of the "community as a whole". The only way to determine what that is, is if people do rate partially on the basis of agreement.


From this perspective, "Karma Whoring" is not a problem, because the basic mode of the karma whore is to define and express the majority view of the community.

IMO the main issues with K5 are different. But that's just my opinion.

The other main purpose of comment rating is to provide an "objective" idea of how much commitment to discussion an individual has, in order to select those who have the highest commitment to good discussion, so that they can be provided the tools to help administer the discussions and keep them high-signal. "Trusted" status is determined by a combination of average rating ("Mojo"), and number of comments contributing to that rating. Trusted users must maintain a high average rating across a reasonable number of comments in order to be trusted.

These criteria should, IMO, be expanded.

This use is the one your view focuses on, and in this sense it is clear that rating according to bias is a terrible thing. In this case, there is no link at all between reflecting the consensus view and being a good contributer to the site. One user may clearly and consistently disagree with the majority view, and be the best candidate for trusted status in the world, but if everyone rates according to agreement alone, they will never be able to assist in the way they should, because they will never become trusted. The danger here is that by rating according to personal bias, only readers who agree with the common view will be trusted, and at some point, they may mistake their trusted status for a general ability to suppress the unpopular view, and start rating comments they disagree with down below the normal threshold.

So, if bias were the only relevant factor, the system as designed is deeply flawed, because it would ultimately lead to a massively bovine state of groupthink, where any dissenting opinion is suppressed. Obviously, we don't want this to happen.

So, the question is, how do we balance these two sides of the coin? On the one hand, some rating should be based on agreement, IMO. It does serve the purpose of clarifying the majority view. On the other hand, some rating must be based solely on the merits of a comment as argued, and not on whether the rater agrees or disagrees.


The hope is that some people will see it one way, some will see it the other, and perhaps some will see it both, or not think about it at all, and just go on gut feeling. On the whole, then, I expect the camps to balance each other, and strike a good balance overall.

This is the "good enough for now" solution, and I tend to be minded similarly.

This may be simply wishful thinking on my part. Your fear is that the bias-voters will overwhelm the objective voters, based on knowledge of human nature. I don't know if this is true or not, and I have seen cases where comments have clearly been rated according to bias alone. I've also seen cases where well-expressed comments that utterly fly in the face of the consensus opinion are rated highly, presumably based on their merits as writing.

I have no objective idea whether one side is "winning" or not. I do know who is trusted, and some of them might surprise you. [Specific user reference deleted - and no, it's not Sig11] Several [...] well-known "dissidents" are also trusted users. I get a daily report of who is trusted, and if anything, dissenting views are overrepresented; i.e. there are fewer dissenters (by definition), but almost all of the known and consistent dissidents are in fact trusted. This is the main reason why I am not yet as concerned as you about the trends in rating.

Good information. Some sort of posting of this to the site might be useful, though I don't know how it would best be accommodated. How do you identify "dissenters", BTW?

However, it is very wise to consider a potential problem before it becomes a real problem. So, are there ways that the rating system could be "fixed" to clarify and perhaps separate it's roles?

I agree with you that one really good possibility is to make the rating system as transparent as the voting system. This has been an idea from the start, and is very likely to appear soon. Having a way to see who rated what comment and what the rating was would, I think, tip the scales in favor of objective rating.

Also opens the door to retaliatory moderating...but, if that increases participation, it might not be all bad <g>.

Bias Solution -- Multiple Moderation Dimensions?

Another idea would be to divorce the two roles entirely. Have two rating choices, one which rates agreement (thus providing the data for the site's "consensus reality"), and the other rating clarity and quality of expression, thus determining who is likely to be a good trusted user.

Ah, you technologists are all the same. Still, the naivete is touching ;-). What's to say that a person wouldn't take the opportunity to knock an objectionable post down twice, or an agreeable one up? You still have to trust the rater. A single-unit metric means the statistic has to be assumed combined, and you're counting on honesty (or proportional representation) to work things out. A "controversy" metric (std. dev.) would also help. Probably immensely.

I'm not sure where you get accuracy. There is no built in means to detect or correct an error in fact.

I believe Rusty was referring to accuracy in measurement, not content. Still, it's a known limitation of K5. Adding a metric for "validity" only sidesteps the problem: you're now assuming that the moderator has the background to make the judgement.

Ultimately, IMO, truth, goodness, accuracy, interest, humor, etc., are emergent within a metric called "value". Typically, an utterly incorrect statement will generate a counterclaim, often with supporting evidence (usually as Web links). As new readers (or those who've already moderated -- you can change your vote) mull the evidence, the truth is taken into account. Ultimately, it would be nice to have recognized experts in a field, but then you'd need to categorize comments by field of appropriateness. Weighing unfavorably on the accuracy v. complexity scale.

And Some Words About Paradise

I think k5 is "lucky" to have such a high signal to noise ratio. Lucky in the sense it was formed by a good team. Rusty and the rest of the cabal, and Scoop, have done a really good job.

But also consider that Slashdot has greatly improved in quality since the majority of detractors have moved to their Shang Ra La at k5.

That's Shangri La, BTW, from Lost Horizon by James Hilton, 1933. Interesting you should bring it up: "Everything in moderation, even moderation".

A Slightly Non-Sequitor Footnote:

Very interested. Have you read Lawrence Lessig's Code and Other Laws of Cyberspace? I'm in the middle of that right now, and it's really helping me clarify how online communities (and K5 in particular, of course) operate,

I read this book about a year ago, it also influenced my thinking in developing the moderation system at K5. Highly recommended. I'll do a capsule summary on request (probably as a K5 post).

Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.

ahem. (1.14 / 14) (#35)
by Signal 11 on Fri Dec 15, 2000 at 01:30:40 AM EST

Eat any good books lately?

Society needs therapy. It's having
trouble accepting itself.
[ Parent ]
I am that K5 contributor (5.00 / 4) (#42)
by maynard on Fri Dec 15, 2000 at 01:49:28 PM EST


Thank you for not revealing my name without consent. I apologize for not contributing to the email exchange more often, but I had a family emergency going on and dropped out of my online pursuits at the time.

I'd like to go over some of the points you made in this post, however, I do agree that the solutions you present do look significantly better than the system now in production. One issue where I disagree is the capitulation of accepting abuse by members. You argue that this is an authentication problem which is unsolvable, though I think that while it may not be possible to prevent users from opening multiple accounts, it should be possible with a database of all comment ratings and users to ferret out gross abuse. Whether such a system of cross references is too computationally intensive to be possible given cost constraints over membership scaling -- well, that I can't answer to.

I disagree with Rusty that ratings should be a measure of personal agreement. I think the FAQ should specifically state that a comment rating should be independent of personal agreement (which is impossible to enforce, I know), and that users should strive for an impersonal assessment of a comment's contribution to a story or thread only. It should be stated upfront to users that the goal is to promote discussion and friendly debate, not a convergence of agreement among the K5 community. JMO

The statistical additions to the rating system are much needed. They will help ferret out individual bad ratings, while showing which comments are poorly rated because of controversy. I agree that even well written iconoclastic views should expect some high ratings given a large enough sample; and conversely, those comments which get rated down by everyone are usually of no value. So, this means that I agree that over a large enough sample a comment tends to get rated far more fairly than from just a few ratings -- especially from ratings of one's opponent. As an aside, it might make sense to prevent rating not only one's own comments, but also those comments in reply.

I want to make it clear that my primary goal of discussing the rating system with you and others was to prevent popular views and people from completely controlling content on the system. It is my opinion that only through deviation of opinion do we have any real content whatsoever. That is, if everyone simply parrots everyone else -- and gets rated up in the process -- then we have no discussion forum to speak of; we'll have monologues written under different account names. So, for the rating system to be successful we need to somehow enshrine in spirit the notion that iconoclastic views are perfectly OK, as long as they're logically formed, well written, and preferably referenced in the factual record. They may even be wrong to the majority, as long as they are polite. I've seen far too many well written posts rated down to below a two simply because they expound views which aren't accepted by a subset of community members. This is particularly the case down a thread, where usually the only members who rate a comment are those who are directly involved in the debate.

Lastly, one thing I didn't see mentioned was how to promote rating by readers alone. I think one can argue that a rating by a reader, rather than a writing and debating opponent, is of greater value to the forum. It might make sense to reward readers with mojo points who also take the trouble to rate. I suggested one way of doing this, by limiting ratings to only those who are not directly involved in writing a particular thread. However, it may make sense to actually give readers mojo points for valid ratings. Of course, you'll need a counter measure to prevent readers from wrongly rating all comments a 1 or a 5... so give mojo when a reader rates comments with a wide deviation from comment to comment.

I want to make one final point. In one of my posts here I personally rated the post down using multiple accounts just to see what would happen. Turns out that over time the post was slowly rated back up to a 3.5... which really does go to show that given enough of a sample the rating system can work. Though I still think that the current production system is fundamentally flawed because of it's close similarity to a prisoner's dilemma paradox. Transparency of who rated what comment could solve that problem.

I believe there was one other participant in this email discussion. I won't reveal his name for the same reason Karsten didn't reveal mine. However, I want to thank Rusty, Karsten, and this other gentleman for a fascinating discussion. I only wish I could have spent more time on the matter. I'm still looking for a free moment to go to the bookstore and pick up Lessig's book. Will do as soon as my life returns to "normal."


Read The Proxies, a short crime thriller.
[ Parent ]

Change of venue (5.00 / 2) (#45)
by Sunir on Sat Dec 16, 2000 at 02:35:12 AM EST

As I stated in the other thread, I'm (unilaterally, sorry) changing the venue of this discussion to MeatballWiki. I will try to consolidate some of the above as well as other points made in the private e-mail discussion.

"Look! You're free! Go, and be free!" and everyone hated it for that. --r
[ Parent ]

Comment Rating is working fine (3.50 / 8) (#34)
by kraant on Fri Dec 15, 2000 at 01:26:18 AM EST

Chin up youngster your comments rating isn't tied to your sexual appeal or your ability to write. And if you think it is then you are taking all this far to seriously.

Back in the good old days (pre-DOS) rating was purely for show. All it changed was the order of the comments if people sorted the comments by rating, which no-one did.

After the DOS attack it was determined that there needed to be some form of co-operative deletion of comments so that the load wasn't all on the admins. And this is where mojo comes in.

Mojo is a very harsh filter because it is far more important to ensure that people who would abuse their trusted status do not become trusted than it is to ensure everyone who wouldn't abuse the privlidge does become trusted.

So I'm sorry if it hurts your ego that you aren't getting high ratings or trusted status. But you are just going to have to learn to deal because the comments that need to be deleted are being deleted fine and therefore the system is working just as it should. And more importantly fiddling with the system so that your poor little self esteem doesn't get crushed could cause the system to cease working the way it should. And that would be a far more painful state of affairs.

As far as rating people by agreement rather than by content I agree that this is not the best state of affairs. But it is a social problem not a technological one and it can only be solved by education. Arguing that this is a reason to get rid of rating is like throwing the baby out with the bath water. One exception to this I would however like to make is for imperative statements where it's far better to have one persons say "This needs to be re-written as it is riddled with grammatical and spelling errors" and have that rated to 5 than it is to see 50 following posts also going "me too" or "I disagree".

Now stop whining


"kraant, open source guru" -- tumeric
Never In Our Names...
I agree it's broken, but this is wrong approach (3.00 / 4) (#40)
by speek on Fri Dec 15, 2000 at 12:39:14 PM EST

Two ways the system is broken - abuse, and non-optimal use by those who try.

To fix the second problem you advise steps like restrict +5 and +1 cause they're overused. Give English descriptions of what each rating means (poor, good, etc). On the contrary, I think the best way to proceed would be to make a note of how people vote, and make your system match how they seem to want to use it.

For example, people use +5 and +1 a lot. I interpret this to mean a binary system would be better. +, neutral, or -. Three radio buttons, neutral the default.

I agree a select box is poor - radio buttons would be better, but let's get rid of some choices. There's no need to rate on a scale of 5. <p. I would also suggest that a response to a post is an implicit +1. Obviously, you thought the post worthwhile to respond to, so.....

Abuse is harder, but I think the abuse is working only because most of us don't rate. I mean, don't vote :-). The only help I can see there is make it simple enough to vote that more people will do so, thus drowning out the abusers.

I'd rather see simpler than more complex.

al queda is kicking themsleves for not knowing about the levees

fixed amount of voting points (none / 0) (#48)
by espo812 on Thu Dec 28, 2000 at 11:37:32 AM EST

How about a fixed amount of voting points per user over a certain period of time (that don't roll over)? Make the points used relative to the current status of the post you are moderating.

For example, if a post is moderated at 3 then to give it a 1 vote costs you two points, but to give it a 4 vote it costs 1 point.

Keep the initial posting status like it is done now (the average of posts over the last week or two).

Just my US$0.02 - feel free to ask for change.


Censorship is un-American.
What's wrong with voting? | 48 comments (44 topical, 4 editorial, 0 hidden)
Display: Sort:


All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!