Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

[P]
The failings of K5 Comment Moderation

By ucblockhead in Meta
Sun Jul 15, 2001 at 06:04:51 PM EST
Tags: Scoop (all tags)
Scoop

The kuro5hin comment moderation system is rightly credited with preventing the sort of crapflooding that plagues slashdot . However, it is clear that the moderation system is suffering from abuses, including mod-storms and a general "moderation inflation" that has made the system mostly useless in terms of evaluating comments. I believe that these problems are a result of the system itself, rather than the failings of its users. The system does not properly take human psychology into account.


Trying to do two things.

The moderation system here at Kuro5hin has two unrelated purposes. The most important purpose is to avoid crap-flooding. The second is to evaluate comments. Unfortunately, these two things often conflict. A zero rating hides a comment. A non-zero rating may unhide that comment, depending on the resulting average. Unfortunately, it is human nature for someone who believes a comment should be unhidden to moderate for a target average rather than the worth of the comment. This often results in virtually worthless comments getting underserved ratings of '5'. These comments may well deserve to be unhidden, but the system forces moderators to undermine the second purpose in order to accomplish the first.

Moderation Inflation

If you believe the FAQ, ratings of '5' or '1' should be rare, while ratings of '3' should be normal. The opposite is the case. The reason for this is that an "average" comment is not going to compell people to take the effort to rate as much as a good comment. As people watch the results of this, they begin to see higher rated comments as "normal", creating a continuing cycle upward. I recently saw someone complain of being "modded down" when he received a '3'.) I believe that we are approaching the point where even a '4' is considered a "low" rating. And as this happens, people start to fear giving out '3's (and soon, I believe, '4's) because they don't want to be perceived as insulting another's post, even though a '3' should not be considered an insult at all. A similar thing is occuring in the downward direction. It is increasingly hard to find a ratings history that isn't a series of '1's and '5's, with nothing in between.

Proposals

Now many will say that the problem is people, and that if people would just read the FAQ and act accordingly, everything would be perfect. Maybe so. But this will never happen because people are people, and they are not going to change. Even if the current members could be convinced to do this, new users will quickly bring back the bad behavior, because the bad behavior is caused by the intersection of the moderation system and human psycholgy. Human psychology cannot be changed, but the moderation system can.

As I said at the outset, the moderation is succeeding at one of its two goals, so we should be hesitant to make severe changes. No one wants things to get worse. However, I think that improvements could be made that would not harm the original intent.

First, comment hiding should be decoupled from ratings. Ratings should be from '1' to '5', with the '0' removed. Instead, trusted users would be able to vote to hide or unhide a comment. A comment would be hidden if over 75% of votes were to hide the comment. This would have a similar effect to today's system, however a vote to unhide would not be a statement of worth. A moderator could vote to unhide and still rate the comment a '1'. This would reduce the number of moderators voting high merely to change the average rating, but it would preserve the existing system, which essentially requires that a high percentage of moderators want something hidden for the hiding to occur.

Second, in order to get a nice bell-curve distribution of ratings, it must be enforced. I propose that a "ticket booklet" system, similar to the way Disneyland used to operate their theme parks in bygone days. Users would still have unlimited moderation ability, but in order to give higher or lower ratings, they'd have to make a number of average ratings. When a user is first registered, they get four "ratings booklet". Each rating booklet includes a single '1', a single '5', two '2's, two '4's and four '3's. The user does not then get another ratings block until at least one of their old ones is used up. In other words, a user would only be able to give out four '1's (one from each booklet) before having to give other ratings. This would make it extremely hard to mod-storm, however, to a user following the FAQ it would result in almost no burden at all. It would also encourage people to rate "average" comments more, resulting in an end moderation that means more. It would also mean that an overall high rating would have more meaning.

Sponsors

Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure

Login

Poll
A comment's rating reflects its worth
o Always 2%
o Sometimes 69%
o Rarely 21%
o Never 5%

Votes: 73
Results | Other Polls

Related Links
o Slashdot
o Kuro5hin
o slashdot
o FAQ
o Also by ucblockhead


Display: Sort:
The failings of K5 Comment Moderation | 85 comments (81 topical, 4 editorial, 0 hidden)
Are scores that important? (4.05 / 40) (#1)
by Carnage4Life on Sat Jul 14, 2001 at 09:08:46 PM EST

*sigh*

I've probably read twenty or more articles on why K5 comment moderation is broken (typically written by people who have a stalker who downrates all their posts) and have been constantly amazed by how people don't seem to get a fundamental point.

K5 comment rating is broken because not enough people rate comments.

Coming up with more complex and convoluted ways to rate comments will reduce the number of regular users who rate comments but won't prevent those with a vendetta against you from taking out their frustrations by scoring your posts lowly.

My advice is to ignore the actions of people who modstorm, in the long run they are just meaningless numbers on a website. marlowe and hotcurry follow me around (I think they are the same person) and mod down all my posts about Open Source and it doesn't do anything but convince of their immaturity.

Ratings scores simply aren't that important. I come here for discussion and typically get whether my post is scored > 4 or < 2 so I can't complain. When my comments score makes me dip below trusted user I hardly notice primarily because I rarely rate comments to 0 or review hidden comments.

I understand... (2.78 / 14) (#4)
by ucblockhead on Sat Jul 14, 2001 at 09:17:02 PM EST

I understand your point, and as it stands, yes, you are right. However, I think the system could be changed so that ratings did have a meaning.

Also, the system I proposed would help prevent people from taking out frustrations with ratings, at least to the extent that they do now. Yes, you could still give someone twenty '1's, but you'd have to also give out nearly as many '5's, and many, many more average ratings in the process. That would tend to blunt people's anger as they'd have to go find positive things before giving out too much negativity.


-----------------------
This is k5. We're all tools - duxup
[ Parent ]

JESSY CHRISTO! (1.91 / 36) (#8)
by Inoshiro on Sat Jul 14, 2001 at 09:35:42 PM EST

However, I think the system could be changed so that ratings did have a meaning.

DO YOU KNOW HOW MUCH WORSE THAT WOULD MAKE EVERYTHING? DO YOU KNOW HOW MANY FUCKING LOSERS WHO RATE WITH A VENDETTA NOW WOULD ACTUALLY GAIN POVER THEN BECAUSE EVERYONE ELSE IS TOO MUCH OF A PUSSY TO RATE COMMENTS? DO YOU KNOW HOW MANY FUCKING IDIOTS WHO CAN'T READ THE FAQ ABOUT RATING WOULD USE THEM AS AN EXCUSE FOR MORE ABUSE?



--
[ イノシロ ]
[ Parent ]
Ummm... (3.50 / 16) (#10)
by ucblockhead on Sat Jul 14, 2001 at 09:48:45 PM EST

Why should we rate comments if ratings have no meaning? You seem to be saying on the one hand that ratings shouldn't have much meaning and yet on the other that we should all be rating as much as possible...

Also, I tried, in the proposal above, to propose something that would explicitly discourage vendettas and encourage more ratings. Maybe you disagree that it would do this. I'd love to here why...


-----------------------
This is k5. We're all tools - duxup
[ Parent ]

mmmm (2.20 / 10) (#33)
by delmoi on Sun Jul 15, 2001 at 05:44:11 AM EST

It's kafkalitious!
--
"'argumentation' is not a word, idiot." -- thelizman
[ Parent ]
Sigh... there are more than two sides. (3.06 / 15) (#35)
by Inoshiro on Sun Jul 15, 2001 at 06:44:38 AM EST

Comment ratings should have no meaning, because then people are less likely to have a bug up their butts about ratings. And you should rate everything, because people get bugs up their butts about comment ratings -- and I get to hear about it because I run the site.

And every few months someone decides that the comment rating system is broken (for whatever reason) and posts about it.



--
[ イノシロ ]
[ Parent ]
what are the others? (3.66 / 3) (#59)
by garlic on Mon Jul 16, 2001 at 10:44:04 AM EST

You said there are more than 2 sides, and then list 2.

The thing I find annoying about comment rating is that it doesn't have any meaning, except for top level comments.

My personal rating of comments doesn't benefit me, but only the people who look at the article after me. So I have to do it solely because I want others to do it as well to benefit me.

But then, it doesn't benefit me that much when others do moderate. I'm one of those people that doesn't look to closely at the headers of comments. I'll catch the title, and maybe the author but thats about it. So then I sort comments based on rating, so I don't have to see the stupid ones. But this effect is negated if I want to also see threaded discussion, since the comments below a high modded comment can be just as low as the comments at the end of the sorting.

Personally, I'd like the ability to hide comments below an individually set level.

But like you said, you run the site. If i am that bitter, I can go elsewhere. If you are that tired of it, you can make it harder to post these stories.

HUSI challenge: post 4 troll diaries on husi without being outed as a Kuron, or having the diaries deleted or moved by admins.
[ Parent ]

More than one model. (1.50 / 2) (#74)
by inpHilltr8r on Mon Jul 16, 2001 at 02:53:10 PM EST

Personally, I'd like the ability to hide comments below an individually set level.

There's several sites that do this, why not try one of those?

[ Parent ]

Actually... (3.00 / 17) (#5)
by ucblockhead on Sat Jul 14, 2001 at 09:18:32 PM EST

Actually, it wasn't a ratings stalker that inspired this. I don't give a crap about them. It was a guy complaining because someone "modded him down" with a '3'.
-----------------------
This is k5. We're all tools - duxup
[ Parent ]
Downmodding (3.33 / 12) (#32)
by delmoi on Sun Jul 15, 2001 at 05:37:21 AM EST

Actualy, the 'trusted user' level is 3.75. So if someone rates you a three, it makes it harder for you to maintain trusted user status.
--
"'argumentation' is not a word, idiot." -- thelizman
[ Parent ]
But (4.00 / 3) (#69)
by Phil the Canuck on Mon Jul 16, 2001 at 01:17:58 PM EST

To be fair, trusted users should be a small minority of the K5 population. Only exceptionally good comments should earn greater than the trusted threshold.

------

I don't think being an idiot comes with a pension plan though. Unless you're management of course. - hulver
[ Parent ]

Gimme my booklet (2.89 / 19) (#3)
by mami on Sat Jul 14, 2001 at 09:12:41 PM EST

I am going to be a serious moderator from now on, I promise.

Please add 6 for "can't rate, don't get it", then I wouldn't rate a 5 too often just to be polite...



oh well, (3.77 / 9) (#36)
by mami on Sun Jul 15, 2001 at 09:04:01 AM EST

the rating of my comments shows there is definitely a 6 for don't get it necessary.

[ Parent ]
Who cares? (3.35 / 14) (#6)
by eean on Sat Jul 14, 2001 at 09:24:02 PM EST

I agree that they should be seperated.


But, in general moderation, while I think is good, is not really all that important. At slashdot there is so much noise (that while mostly are trolling or spam, there are a lot of stupid comments) that moderation is important. At Kuro5hin it just encourages good behaviour. I'm not really convinced that scoring polarization is so much a problem that some sort of limited scoring needs to be put in place. Perhaps it is something to think about as Kuro5hin grows in size, but now I do not see it as a problem.

Even easier (3.84 / 25) (#11)
by FlinkDelDinky on Sat Jul 14, 2001 at 09:58:27 PM EST

How about we drop the numbers from the rating system and use words instead

1 = Offensive
2 = Poor
3 = Average
4 = Good
5 = Excellent

In addition all unrated posts will be listed as Average. Also the rating (3.66 4.5 etc) will not be listed, only the nearest word will be listed, as such 4.5 will list as Excellent while a 3.4 would be listed as Average.

Do you see how incredibly clear and easy that is to understand from userland? Christ you wouldn't even have to read FAQ to figure it out. Glory be, could K5 actually have a rating system that could be comprehended without having to read a FAQ?

1 shouldn't be offensize (3.33 / 9) (#18)
by eean on Sat Jul 14, 2001 at 10:56:37 PM EST

Moderation shouldn't be done on the basis on if they are offended by posting. I know that this isn't the case, which is why I think the whole conversation is a little silly because it never will be.

Instead of Offensize perhaps "Not worth the bandwidth."

[ Parent ]
Rating comments sucks. (4.05 / 36) (#12)
by Defect on Sat Jul 14, 2001 at 09:59:17 PM EST

Rating comments is a freaking chore. It's not something where i wake up in the morning and shout to the world "Alright Defect, let's go rate us ten-teen billion comments."

Rating any more than one or two comments at a time every once in a while is a pain in the ass. Don't get me wrong, i like the concept, i like how it's open, i'm just a god damned lazy son of a bitch, but the problem is, i think most people are god damned lazy sons of bitches. I say this with confidence because 10 ratings on a comment is considered a good amount, but there are over eighteen thousand users on this site.

If comments were somewhat easier or more rewarding to rate, more people would rate and this whole booklet system would be completely unnecessary, as the more ratings a user gets, the more accurate the total score is (as if we haven't gone over this enough before). I have no idea what i mean by "rewarding" and that most likely could be an entirely different story in itself. And hell, for that matter, i have no idea how rating comments could be much easier either.

And on the hidden comment issue, i like your idea but i think it will still have just as many problems as the current implementation. First off, with the way you propose it, a trusted user could go through and hide a load of comments, and regardless of what their ratings were beforehand, they would all become hidden until other trusteds went through and voted to "sjow." The way it is now, rating a comment '0' doesn't necessarily hide it if the score is already above 1, which, as has been proven, is a very good thing, as "0-storming" has been fairly active as of late...
defect - jso - joseth || a link
I'm a god damned lazy son of a bitch (3.37 / 16) (#45)
by speek on Sun Jul 15, 2001 at 06:18:57 PM EST

I give only 1's or 5's, cause I don't want to think about 2's, 3's, or 4's. Binary is the way to go. That's one step toward making it easier to rate.

Another step - use radio buttons rather than a select. A radio button is one click. The select is 2.

Submit all ratings regardless of which link the user clicks next. In other words, let the user rate and forget - no more having to hit "Rate All" before responding to a comment, or going to the next story. (Actually, I don't have too much trouble with this - I hit "Rate All" and then immediately hit "stop" on my browser. It works).

Yes, these changes must be implemented for all lazy sons of bitches like myself. Or not.

--
al queda is kicking themsleves for not knowing about the levees
[ Parent ]

One more suggestion (4.00 / 5) (#63)
by alder on Mon Jul 16, 2001 at 11:56:30 AM EST

Why does [Rate All] take us back to the discussion we are about to leave? Maybe this is derived from my personal habits, but on the rare occasion when I rate comments, I click the button after reading all of them, and my "natural" exectaion in those cases is to be returned to the "previous" page - a "front page", a "section".

Sometimes I put ranking on some especially good comments, but in the end never press the [Rate All] button. I just want to move forward, not to wait until this, which is already a history at the moment, page is reloaded again, just for me to click somewhere else to return to where I was before...

[ Parent ]

Why is it difficult? (3.16 / 6) (#67)
by Elkor on Mon Jul 16, 2001 at 01:12:23 PM EST

You read a comment, select a number from the drop down box, move on to the next comment.

At the bottom of the page, you click on the "rate all" button.

If you have trouble figuring what to rate a comment, don't rate it at all.

Or am I missing something?

Regards,
Elkor


"I won't tell you how to love God if you don't tell me how to love myself."
-Margo Eve
[ Parent ]
where did i say it was hard? (2.50 / 2) (#76)
by Defect on Mon Jul 16, 2001 at 05:14:41 PM EST

It's just a pain in the ass for no apparent personal gain. I'm selfish and lazy, i never said rating comments was rocket science.

Take that mindset, plus the idea that the only rating that is going to make anyone in the world happy is a high one, then rating every single comment seems worthless. When you rate someone low, no one at all actually benefits, the only thing you've accomplished is upsetting someone you've never met before.

Seriously, what's the point?

I just tried rating every comment in a story, and i didn't get a damn thing out of it. I didn't even feel happy afterward because as soon as i hit "Rate All" and the page loaded anew, there were more comments to be rated. It's like your mother telling you to clean your room, and as soon as you think you've done a good job she kicks you in the nuts and empties a garbage can in the middle of your floor. There's no end and no reward.
defect - jso - joseth || a link
[ Parent ]
Not for everyone... (3.00 / 5) (#77)
by mcherm on Mon Jul 16, 2001 at 05:24:41 PM EST

Well, maybe you and I just think about it differently. My favorite thing to do with K5 is to browse it and moderate the comments. I feel like I'm contributing to the discussion that way, by helping to pick out and hilight the really good stuff, without having to write comments myself. And it improves the signal-to-noise ratio.

But I *DO* have a problem with the ranking-inflation that I see here. Hence, my sig (I've just re-directed it to this article!).

-- Michael Chermside
[ Parent ]

One solution (3.38 / 18) (#13)
by slaytanic killer on Sat Jul 14, 2001 at 10:12:38 PM EST

I agree with Inoshiro's earlier rant that this would complicate the system enough that less people would moderate.

However, one solution may be to make the user interface better. I've forgotten all my Javascript and most of my HTML... but is it possible to simply send a person's ratings to the server whenever she clicks on any K5 link? If a user selects a score under a comment from the drop-down box, couldn't it be sent to the server after clicking on a normal link?

This does not have the problem of errors, since the user can always go back and change the rating. And she can even now see her own rating history.

Javascript popup (3.30 / 10) (#14)
by nebby on Sat Jul 14, 2001 at 10:18:28 PM EST

The way I do it for ratings on half-empty is a javascript popup that goes in the background. The alternative is to have a hidden frame, but this mangles the URL.

Half-Empty: A global community of thoughts ideas and knowledge.
[ Parent ]
Making modding more convenient (none / 0) (#85)
by buzco on Fri Jul 20, 2001 at 05:24:00 AM EST

I've forgotten all my Javascript and most of my HTML...

Forgetting (or never learining) JavaScript is a good thing, too bad you forgot most of your HTML.

but is it possible to simply send a person's ratings to the server whenever she clicks on any K5 link?

Unfortunately, no. Forms buttons are entirely separate from normal links. What could be done would be to provide a text box with the "rate all" button that identified the next link you wanted. One could then simply drag the link to the box, then click on the "Rate and go" button which would have the desired effect.

If this last could be implemented without making things too cumbersome for the user or cluttering up the screen too much, I think it would be a good idea. In fact, I think the idea so good that, if Scoop were in PhP instead of (ugh!)perl I might just submit the code for it myself!

If a user selects a score under a comment from the drop-down box, couldn't it be sent to the server after clicking on a normal link?

This is essentially the same as the above.

What could be done with JavaScript is to send the data to the server every time one selected an item on the pull-down.

This would have the severe disadvantatge that then people that didn't have a JavaScript capable browser with JavaScript enabled (with all the attendant vulnerability to trojans and viri) would be unable to mod at all. Alternatively, the code would have to be modified to have JS and non-JS versions (so that everyone could vote and those that didn't mind the risk could have the convenience).



[ Parent ]
The average is bad, m'kay. (3.83 / 24) (#15)
by wnight on Sat Jul 14, 2001 at 10:22:37 PM EST

The big problem with the current scoring system is that you have to cheat to get the results you want.

Currently, if you see a post with +5 (2 mods) and you think it belongs at 3, you have to vote 1 to drop the score even close (3.6) to where you believe it fits. If you vote your desired result, the score only drops to 4.3 which isn't your intention.

As I see it, we need to use the mode, not the mean.

Using the mode will encourage people to score an article as they wish it to be seen, not in the manner most likely to manipulate the score close to where they think it should be.

One huge benefit of using the mode is that stray +1 or +5 ratings, either from an enemy, or friend, are lost, under the weight of the non-biased voting. Instead of a +1 vote dragging down a highly-rated post, it's completely ignored, until ratified by having a significant number of other voters agree with it.

Types of averages, mean, mode, and median.

Dictionary.com says ..

Q.: What is the difference between average, mean, median, and mode?

A.: Mean is one kind of average. It is computed by summing the values and dividing by the number of values. Two other common forms of averages are the mode and the median. The mode is the frequently occurring value in a set. The median is the middle value of the set when they are ordered by rank.

Average is a synonym for arithmetic mean -- which is the value obtained by dividing the sum of a set of quantities by the number of quantities in the set.

An example, with 10 votes, will show how the post would be scored in each case.

Votes: 1, 1, 3, 3, 3, 3, 4, 4, 5, 5, 5

Mean: 3.2 (32 / 10 - This is the usual "average")
Mode: 3 (3 is voted for 4 times)
Median: 3.5 (The "middle" value in the list 1,3,4,5 is between 3 and 4, thus 3.5)



Mode not the way to go (3.88 / 18) (#20)
by nads on Sat Jul 14, 2001 at 11:13:08 PM EST

The mode is definitely not the way to go. For instance let's say 6 people rated the parent of this comment in the following manner. Two people gave the comment a 1, and four other people gave the comment a 2,3,4, and 5 respectively. The mode of the scores is 1. Although the majority fo the people definitely think the comment is better than a 1. The average is 3.2 which definitely seems more indicative of the comments worth. The median also has problems. Perhaps using a combination of the average and standard deviation to rate comments would work. Where a lower standard deviation equates to a better rating.

[ Parent ]
Point. How about... (3.55 / 9) (#23)
by wnight on Sun Jul 15, 2001 at 12:51:52 AM EST

Ok, so the mode alone has issues... I didn't think that was as bad, because the chance of a few people down-modding something seemed fairly low (how many people dislike you and follow you around) when the real opinion is that you deserved a higher score.

But, it could happen. Your idea of the average, with error-checking via the a standard deviation, seems like a good idea.

How about taking an average, discarding everything too far from that, and taking the average again? This way, anomalous ratings are ignored.

I think I'd have also started posts at 1.5 or so. This way someone can mark them down and they sink, a bit, but even one reasonably high rating (3 or more) will raise them, despite the down-mark.

I like the idea that posts mostly rise up from the pool, instead of sink out of sight. It makes people concentrate on positive modding instead of negative. But... some posts really need to be dropped a bit, without deserving being hidden. It'd be like modding now, except that there'd be a bit of downward influence to use. This is less dangerous than it would be on Slashdot with limited moderation points... Here, if you see something you think is just a little high or low, you can do something, without missing the chance of modding something really deserving later. So if anyone drops something unfairly, someone will likely correct it.


Or, we just resort to a little meta-moderation. If someone mods five posts by someone else, out of eight or less, say, and the votes are all the same (high or low) then it gets flagged and a trusted user gets to see it, if they think it looks unfair, an admin looks at it, and deletes an account if it's warranted. (ie, if the posts are in multiple threads, and if this person did little else other than mod this one person, as opposed to someone who's just a busy modder.)



[ Parent ]
Correct (3.12 / 8) (#27)
by Nyarlathotep on Sun Jul 15, 2001 at 03:16:57 AM EST

Yes, the mode will not work with the current system, but if you reduce ratings to Good (1), Average (0, default), and Crap (-1, trusted user only) then you get all the advantages of the mode but still get to keep your average. I'm not shure the 1-5 scale really expresses anything anyway, but instead of tring to force people to express things with a 5 point scalelike the article suggests then you should just simplify to the system people really use.

Campus Crusade for Cthulhu -- it found me!
[ Parent ]
On the value of the 1-5 scale (3.50 / 2) (#78)
by mcherm on Mon Jul 16, 2001 at 05:36:26 PM EST

Well, *I* personally find the 1-5 point scale FAR more expressive than a binary vote. I'd probably stop voting completely with a binary scale.

-- Michael Chermside
[ Parent ]
Mm'kay Mr Macky (4.14 / 21) (#25)
by NightRain on Sun Jul 15, 2001 at 01:38:38 AM EST

Currently, if you see a post with +5 (2 mods) and you think it belongs at 3, you have to vote 1 to drop the score even close (3.6) to where you believe it fits. If you vote your desired result, the score only drops to 4.3 which isn't your intention.

What you're saying is that it's broken because people automatically assume that their own vision of what a comment should be rated at is more important than what everyone else thinks?

The fact is that if you think something is worthy of a 3, then vote it 3. If other people disagree and vote it at 5, then the average will be higher than 3 and that's the way it should be. You shouldn't be voting to try and bring down the average, as that's not what it's about. You are meant to say 'this is what I think of the article' and let the average be exactly that, the average. You don't have the right to determine the average by yourself.

Don't vote, it only encourages them!


[ Parent ]
It's broken (3.44 / 9) (#50)
by roystgnr on Sun Jul 15, 2001 at 11:34:11 PM EST

What you're saying is that it's broken because people automatically assume that their own vision of what a comment should be rated at is more important than what everyone else thinks?

It's broken because people like me who do as you suggest and rate comments the way we believe they should end up do not influence the final score as much as people who abuse the system. A system which "punishes" people for doing the right thing, when such punishment is unnecessary, is broken.

Someone else already pointed out why scoring posts based on the mode would be flawed, and the idea of a "score/deviation" report would be needlessly complicated, redundant (since anyone who cares that much can see the complete score breakdown), and wouldn't fix the problem. The best solution is probably to give posts their median score. I dislike the loss of resolution that would bring, but it's not like most posts get enough ratings for fine-grained averaging to give us three significant figures of "accuracy" anyway.

[ Parent ]

Summary (2.50 / 2) (#82)
by timmyd on Tue Jul 17, 2001 at 08:31:40 PM EST

I think this guy notices that people want their vote to count more than other peoples. So when I want to make a difference, I either vote 1 or 5 depending on whether i want it to go down or up. The problem isn't that people need text representations instead of numbers, they just need a better way to calculate it.

[ Parent ]
Want to make a difference? (4.00 / 2) (#83)
by wnight on Wed Jul 18, 2001 at 05:47:34 PM EST

Yeah. That's basically it. People want to make a difference, not just vote and have it be ignored.

If they think they can make the biggest change by bending the rules, they will.

We need to realize this and design systems that aren't as open to it, or that provide a dis-incentive to fudge the numbers.

Currently people tend to vote +1 or +5, to make the biggest difference. If we can make this backfire, we'll train people out of it.

If a strict average system, voting to make the most difference makes sense. In a mode-based system if you vote for anything other than your desired result, you run the risk of getting it.

For instance, you see a post ranked at 5, you think it should be a 4. Currently you'd look at how many other people voted and at 2 or more, you'd vote 1, to drop the result to 3.6. (Or higher, depending on the number of votes.)

Ideally you'd vote 4, making either no difference, or if you tipped the balance, modding the post exactly to 4.

This does need some safeguards to make sure the mode isn't far outside of the average. If nine people vote 3,4,5 (three each) and four people vote 1, the mode would be 1. As another poster responded to me saying, this isn't the desired result. If we average the score, cull anomolies, and then take the mode, it gets better results.



I think one other thing that should be fixed, and has really irked me, is how a post with score 5 from one vote is ranked above one with score 4.8 from five votes.

Someone else in the thread mentioned making posts start at 3, with a 1-vote weight. This makes 1 and 5 the unreachable limit, a post with more votes gets closer to this limit. With this method, the above posts would be scored 4 and 4.5 respectively.

I'd recommend starting posts at 2, so than people were rarely modded down (which is the cause of most mod-related venom on Slashdot). Instead of this, people would concentrate more on modding each other up.


But yeah, it comes down to me realizing that people will bend the rules when possible. Any anonymous system that wants to survive needs to have rules that can't be bent in your favor.


[ Parent ]
One good suggestion (3.70 / 17) (#16)
by Skippy on Sat Jul 14, 2001 at 10:28:38 PM EST

I like your suggestion that hide/unhide be separated from everything else. The "ticket" idea, however, sucks.

I'd like to see all comments start with a default score of 3 and a weight of 1 moderation. This would make it obvious that 3 is average and moderation doesn't change other than 5 and 1 now become mathematical limits.

# I am now finished talking out my ass about things that I am not qualified to discuss. #

Detminents of downmodding (3.66 / 12) (#24)
by wnight on Sun Jul 15, 2001 at 12:58:07 AM EST

If the system moved to a different default score, I'd think it should be 2...

This way if posts are rated fairly, only the very worst are modded down, everything else gets modded up, or left alone.

From a purely technical/rational view, this is nearly the same, but from a human point of view, this encourages modding up a lot more than down.

Slashdot is a very negative place towards moderators because a lot of facist jerks run around modding people down. If a down-mod was almost never deserved, they'd have a harder time justifying this.

I think we should weight the system to where you almost always elevate a post above the default, rather than push it down below.

I like this idea though, of how K5 sorts posts by score but doesn't support a threshold. This way default-level posts get seen if you scroll far enough, or if they're a response to something. Many posts don't ever get modded, so it's good that they're not hidden arbitrarily.


[ Parent ]
Ratings booklet (3.45 / 20) (#17)
by jasonab on Sat Jul 14, 2001 at 10:40:26 PM EST

First, I like the hide/unhide idea. I know rusty and I have said in the past that the 0/5 war balances out, but the hide/unhide vote makes more sense.

As to the booklet, it's interesting, but unworkable. I think it will just encourage people to give random ratings to various comments so they can get more "useful" numbers.

As others have said, not enough people rate comments. I have to admit to being somewhat guilty of this myself. I rate comments I really like or really don't like. The '3' comments tend to be ignored.

In the end, I have to say I don't find ratings to be particuarly useful, other than to judge what the community thinks of a particular comment. K5 doesn't allow filtering by comment level, and even if it did, the ratings are too arbitrary for me to do that. Too much liklihood of a good comment being rated down by people who disgree with it.

I'll say what I said at Slashdot two years ago: the moderators should be a small group of dedicated mods. I was one of them when Slashdot went to that system, and I argued for keeping it when /. moved to the current system. Let only the trusted users moderate comments, continuing to promote and demote as we do now. At least then we would know that the people rating comments had some concept of how the site should work.

It would form a clique. (3.33 / 6) (#56)
by pin0cchio on Mon Jul 16, 2001 at 03:17:50 AM EST

Let only the trusted users moderate comments

Which means the trusted users elect new trusted users. Those who follow politics know the dangers of an appointed governmental body that appoints its own members with no voter input. (We also know the dangers of anonymous voice votes with no voter input, but that's another comment.)

At least then we would know that the people rating comments had some concept of how the site should work

If this is your goal, try something similar to the system that Everything uses: 25 good writeups gets you voting powers, and you get a limited number of votes per day.


lj65
[ Parent ]
A better idea would be to make the system simpler. (3.89 / 19) (#19)
by Xeriar on Sat Jul 14, 2001 at 11:08:55 PM EST

Like make the comment rating into a radio button list by the box, while utilizing the other suggestion to give names to the ratings, as:

1 = Trash
2 = Poor
3 = Average
4 = Good
5 = Excellant!

And trusted users can also have a radio button to hide a comment, as you suggested, though I think 50% would be better.

Even clearer would be a system in which your average rating actually set the balance - that is, if you tend to rate things as 2's all of the time, your two becomes a three when weighted. Rating things as ones and fives all of the time makes them worth twos and fours, etc. What I'm saying is that, things can be as complex as necessary only when hidden from the user.

----
When I'm feeling blue, I start breathing again.

What is....? (3.36 / 11) (#22)
by Andrew Dvorak on Sat Jul 14, 2001 at 11:55:20 PM EST

The problem with assigning the names to the numbers is that I would then have to ask myself "*WHAT* is excellent", "*WHAT* is Good", etc.. With just numbers (the current system), I can define my own meaning to numbers with the understanding that the ratings are relative, where 1 is the lowest and 5 is the highest.

Defining the numbers, no matter how fuzzy, would curve the results even more, I would imagine, subliminally.



[ Parent ]
"Trash" already has a number assigned to (2.80 / 5) (#39)
by ti dave on Sun Jul 15, 2001 at 03:47:35 PM EST

A "Zero".

So why would we have redundant comments attached to the individual scores?

Cheers,

ti dave
"If you dial," Iran said, eyes open and watching, "for greater venom, then I'll dial the same."

[ Parent ]
Zero Rating (2.80 / 5) (#49)
by MrAcheson on Sun Jul 15, 2001 at 11:26:48 PM EST

A zero is applied to spam. Period. Nothing else gets a zero. If it has even a twinge of content or applicability it should get a one.


These opinions do not represent those of the US Army, DoD, or US Government.


[ Parent ]
Really? Can you explain why... (3.00 / 4) (#53)
by ti dave on Mon Jul 16, 2001 at 01:13:31 AM EST

Gisano gets nearly all of his comments rated zero?
Looking through the lot of them, he occasionally submits a kernel of truth.

Cheers,

ti dave
"If you dial," Iran said, eyes open and watching, "for greater venom, then I'll dial the same."

[ Parent ]
Spam isn't defined in the FAQ... (3.00 / 4) (#55)
by ti dave on Mon Jul 16, 2001 at 02:49:00 AM EST

So how do you interpret the difference between "Spam" (0) and "Noise" (1)?

Is a "Spam" comment an "Extra Noisy, with Noisy sauce" comment?

I have my interpretation of "Spam", and if I were to use it in a *literal* manner, any post with a reference to someone's web project would receive a "0" from me.

I've mentioned this before, I tend to give zeroes to comments that are pure flame, with excessive profanity.

Cheers,

ti dave
"If you dial," Iran said, eyes open and watching, "for greater venom, then I'll dial the same."

[ Parent ]
Spam (3.75 / 4) (#73)
by MrAcheson on Mon Jul 16, 2001 at 02:48:56 PM EST

Spam is a comment with no topical content. It could be a plug for a website with nothing to do with the discussion. It could be a profane attack on someone. It could be completely off topic content.

For instance I will rate the following comments to zero: "Come to website X for hot girls." "You are a $%&@ idiot." "Please help me, Inoshiro won't stop raping my butt." "Natalie Portman, Hot grits, AYBABTU."

However, if you say something like "You are a $%&@ idiot for believing XYZ." then I will give you a 1 provided XYZ has to do with the current discussion. Why? Because you are expressing an opinion in the discussion. You are contributing, but not well or coherently. If you say "You are a $%&@ idiot for believing XYZ because..." you will get more depending on how well you show faults of the original posters reasoning.

The point is that spam is a post which lacks any contribution to discussion and is therefore superfluous and should be removed.


These opinions do not represent those of the US Army, DoD, or US Government.


[ Parent ]
I like your style! (3.50 / 4) (#75)
by ti dave on Mon Jul 16, 2001 at 04:35:43 PM EST

But if you scroll back, in the Hidden Comments section, before the multiple "test" comments, you'll note that many comments fitting this bill:

"You are a $%&@ idiot for believing XYZ because..."

Have received zeroes.
I wish more people agreed with your moderation style.

Cheers,

ti dave
"If you dial," Iran said, eyes open and watching, "for greater venom, then I'll dial the same."

[ Parent ]
Ratings booklets are bad... (3.86 / 22) (#21)
by Anonymous 6522 on Sat Jul 14, 2001 at 11:23:39 PM EST

...and here's why: It assumes that there are always going to be a certain number of 5 comments, a certain number of 3 comments, etc., and that they all will be distributed evenly through time. Let's say you find an excellent thread with a bunch of comments that all deserve 5s. You wouldn't be able to rate those comments properly unless, for each comment in the thread, you go out and fulfull your quota, by finding comments that deserve the other ratings or by just rating randomly. Comments aren't produced to fit nicely into a bell curve, and trying to shoehorn their ratings into one won't make things better.

Now for ratings inflation. A 3 really is being "rated down." There is exactly one concrete thing that you can get with high ratings, and that's trusted user status. It is awarded when your mojo is 3.5 or higher. Now, it doesn't matter how important it is, it's a sign of status to some people. A rating less than 3.5 is bad, and a rating greater than 3.5 is good, and for many nothing in the FAQ says is going to change that.

Maybe the status is the key... (3.75 / 4) (#64)
by alder on Mon Jul 16, 2001 at 12:08:37 PM EST

I tend to agree with your suggestion:
Now, it doesn't matter how important it is, it's a sign of status to some people.
So, in spite of the fact that it - the trusted status - serves a purpose, and probably useful, maybe it brings more bad with it that good. And maybe for the sake of helth and quality of discussions it's better to get rid of it...

[ Parent ]
Current system is excellent. (3.31 / 16) (#26)
by i on Sun Jul 15, 2001 at 02:53:37 AM EST

It works. It filters out trash. Everything else is meaningless. If you want to put a separate filtering system in place, that's fine, as long as it works. With a filtering system that works, the site can go fine without any rating system whatsoever.

and we have a contradicton according to our assumptions and the factor theorem

What...this hasn't died yet? (3.54 / 22) (#29)
by psctsh on Sun Jul 15, 2001 at 04:52:00 AM EST

I've been thinking about this for a while, and how about this for a rating system:
  • Rating 0 - Spam
  • Rating 1 - Inane/noise comment
  • Rating 2 - 'Normal' comment/slightly Inane
  • Rating 3 - Good point/neat reply
  • Rating 4 - Great point
  • Rating 5 - Pure Gold
And then, and alternate way to view these ratings would be to think of a "3" as a normal response. A comment that is not as good as a "3" could get either a "1" or a "2." A comment that is better than this however could be given the value of "4" or possible even "5." A good point to note is that, should this system be implemented, one shouldn't get upset over a score that's not perceived to be "high enough"--mainly because it allows leeway for individual interpretation.

I dunno, what do you guys think? Maybe I should submit this as a suggestion for scoop? Maybe a meta-article?

Oh for god sakes (3.08 / 24) (#30)
by delmoi on Sun Jul 15, 2001 at 05:06:59 AM EST

The current system works fine!

Kuro5hin gets less commets dropped that front page stories.

Who cares, god. A comment's score hardly matters at all!


--
"'argumentation' is not a word, idiot." -- thelizman
Thank you! (4.00 / 7) (#52)
by mauftarkie on Mon Jul 16, 2001 at 12:55:56 AM EST

Who cares, god. A comment's score hardly matters at all!

I was beginning to wonder if I was the only one that thought this whole article was, to be blunt, pointless. Maybe I'm alone on this, but I don't care about comment ratings. At all. I don't even look at comment ratings. It's faster for me to stop reading what I deem a pointless comment and move on than it is to sit there (or as some do, whip out my 40 fake accounts), and rate it a 1.

On one hand, I like voting on the story submissions, because that gets rid of the chaff. On the other, all the user comment ratings seem to do is start pissing wars that make the whole site look like it's frequented by a bunch of juveniles. Maybe that's okay for you, but Kuro5hin has a helluva lot of potential, some of which is really starting to shine, and I will be very sad if it goes to the wayside because it's torn apart by some uppity, whiny users who can't get along. Now I don't want to see everyone agreeing and kissing derriere; that would be boring. I would love to see the mutual respect of different opinions, and the intelligent conversation that goes with it. I think we all do.

If this problem just can't be ignored, then I put forth the suggestion of trusted moderators. This is hardly an original idea. Furthermore, have moderaters suggest potential moderators, and hold weekly votes. Open moderation, sadly, just leads to the abuses we're seeing -- it's just human nature to try and screw the system. The more you try to fix it, the harder people will try to break it.

I remember reading a comment once, might have been here, might have been /., that stated something to the effect of, "You give people a metric, and they'll compete against each other." Do we really want that kind of competition here? I'll be the first to say, I don't; but that's exactly what I'm seeing.


--
Without you I'm one step closer to happiness without violence.
Without you I'm one step closer to innocence without consequence.


[ Parent ]
What if we just keep it simple? (3.50 / 4) (#66)
by alder on Mon Jul 16, 2001 at 12:34:56 PM EST

You are absolutely right - metrics atracts competition. The more points of ranking exists the more places in the system they introduce to compete, including screwing the system...

When I first came to K5 the queue just appeared, and there were no comments ranking. Who knows, maybe because there were no metrics, at least obvious, the only reason to come here was to read good stories, interesting comments, and share the grain of knowledge or ideas with the rest of K5.

Unfortunately, it appeas that recently some of the comments, and on some stories a larger part of them, are there just as handles to gain a "score". That makes some discussions, even if you skip though "empty" comments, much less attractive.

I guess, a longing for better discussions, better stories makes people suggest "imporovements" to moderation system - to promote what is good here for others to see and follow. Maybe this is also a wrong approach? Maybe the best way is just not to attract people who do not come here for information, but see K5 as just another playground for a conceit. Maybe we need to get rid of comments moderation?...

[ Parent ]

TWO ratings: Quality, Agreement (3.47 / 17) (#34)
by snowlion on Sun Jul 15, 2001 at 06:25:39 AM EST

I wish there were two ratings, one for quality of the posting, and one for whether we agree or not.

Face it, a lot of the time we post, we do it to say, "I agree", or, "I disagree," or even, "I think your opinion sucks", or, "Way to go! High 5!"

We can just abstract that process into "Strongly Disagree", "Disagree", "Neutral", "Agree", "Strongly Agree".

There you go. The K5 Popularity score. It'd make me happy, and make it less likely that people would vent through the quality score.

Incidentally, the quality score would be something like "SPAM", "Poor", "Moderately Interesting", "Well Argued", "Excellent Point", or something like that.


--
Map Your Thoughts
Problems (3.62 / 8) (#40)
by Elendale on Sun Jul 15, 2001 at 03:55:03 PM EST

The thing i dislike about renaming the comment moderation is then you get something like i do on slashdot:

"Hmmm, well this comment isn't terribly informative, not insightful, not funny, not underrated, where's the '+1, not a complete pile of crap' moderation option?"

or

"Why can't i rate comments '-1, completely fucking stupid' or '-1, surprised the poster is even literate' anyway?"

Both of which, by the way, are actual remarks.
More than that, i don't think it will actually solve the "problem" of comment ratings. Technical solution to social problem and all that jazz.

-Elendale
---

When free speech is outlawed, only criminals will complain.


[ Parent ]
Quality & Agreement system, Tech solutions to (3.57 / 7) (#41)
by snowlion on Sun Jul 15, 2001 at 05:06:20 PM EST

In the system I gave, "Isn't terribly informative, not insightful, not funny, not underrated; the +1 not a complete pile of crap" would be classified as "Poor" in the *quality* section, rather than in the Agree/Disagree system.

I quite disagree that technical solutions to social problems don't work, there's too much evidence against it. Even though you may not like the Slashdot moderation system, I find that when I say, "Show me only the score 5's", I manage to get a few interesting (at least) posts, rather than, say, ads for X10 or viagra. Without the mod system, that would be impossible.

K5 itself is a solution to a social problem: Finding interesting people. It allows me to find lots of interesting people through the use of special techniques/technology, namely the division of topics, the ability to look up posters, and all sorts of other features.


--
Map Your Thoughts
[ Parent ]
I sort of agree. (4.20 / 5) (#54)
by Kaki Nix Sain on Mon Jul 16, 2001 at 02:21:42 AM EST

I mentioned something like this in someone's dairy, rusty spake thus.

To keep down abuse by using both channels to communicate the same thing, quality or agreement, one could pull a cheep trick and only let someone use one or the other for each post. But that sucks.



[ Parent ]

I have another, simpler, idea (3.65 / 20) (#37)
by boxed on Sun Jul 15, 2001 at 11:17:19 AM EST

To take human psychology into account I suggest a simple relabling. Instead of 1 to 3 I suggest something along the lines of this:

1
2
normal
4
5


This would point out that 3 is normal in a much more obvious way.

I have another idea that might help (3.83 / 18) (#38)
by boxed on Sun Jul 15, 2001 at 11:24:31 AM EST

Use the same rule for comment ratings as for submission queue "ratings": The rating is not shown unless you have voted. This has the strong advantage of forcing people to mod as they see fit instead of "correction modding" where you aim at a certain end score.

I don't like this idea (3.22 / 9) (#43)
by jonnyq on Sun Jul 15, 2001 at 05:46:13 PM EST

There is a key difference between the moderation of stories vs. the moderation of comments. While I do admit that this would indeed be the fairest solution to each comment, it would completely eliminate the usefulness of the moderation for me, and likely a lot of other users.

In voting on a story, it is important to judge it based on its merits and whether you think it will generate a good discussion rather than what the prevailing opinion is. On a comment though, it is almost the opposite. I want to know if people think this comment is insightful or full of hot air, even if I am not qualified to make the judgement myself.

[ Parent ]
some quick fixes.. (3.33 / 12) (#42)
by rebelcool on Sun Jul 15, 2001 at 05:30:02 PM EST

i think that in the rating box itself there should be a description next to the number. I think alot of newbies feel that "1" means "really good" while "5" means the opposite (ive seen a number of postings of mine rated highly by well known users, but receive strange 1's from people who've never posted a comment...I suppose it could be intentional, but i doubt it)

I think ratings should be restricted to people who have posted a comment or two that are not hidden. This would prevent one-time new accounts from being made, and help lessen abuse of the system.

COG. Build your own community. Free, easy, powerful. Demo site

Mysterious 1 ratings (3.57 / 7) (#44)
by Carnage4Life on Sun Jul 15, 2001 at 05:57:30 PM EST

(ive seen a number of postings of mine rated highly by well known users, but receive strange 1's from people who've never posted a comment...I suppose it could be intentional, but i doubt it)

Those are probably from modstormers who are using a throwaway account to mod you down, safe in the knowledge you or others can't mod that account down to untrusted status since it has no comments to mod down.

[ Parent ]
yep (3.00 / 8) (#46)
by rebelcool on Sun Jul 15, 2001 at 06:34:42 PM EST

which leads to number 2 on my recommendations

COG. Build your own community. Free, easy, powerful. Demo site
[ Parent ]

moderation (3.00 / 5) (#61)
by garlic on Mon Jul 16, 2001 at 11:07:46 AM EST

an interesting idea. It could be implemented something like for each comment in the last week, you get 10*(score - 1) moderations available to you. It promotes discussion.

Unfortunately, it also promotes the multiple account owners to use their good account to mod up their bad ones.

The only thing I can think of to try to get rid of multiple accounts is to track IPs, and the activity that those accounts with the same IP do. I don't know how easy or worthwhile masking or faking your IP is though.

HUSI challenge: post 4 troll diaries on husi without being outed as a Kuron, or having the diaries deleted or moved by admins.
[ Parent ]

Musings on multiple accounts (3.50 / 4) (#62)
by mauftarkie on Mon Jul 16, 2001 at 11:55:40 AM EST

The only thing I can think of to try to get rid of multiple accounts is to track IPs, and the activity that those accounts with the same IP do. I don't know how easy or worthwhile masking or faking your IP is though.

If you can figure out how to reliably track through NAT'd firewalls, this could be an option. I'm sure my roommates, who also frequent this site from home, would hate to stop visiting because Kuro5hin somehow thinks we're all the same person and starts denying some of us access.

The only way I can think of to absolutely get rid of multiple accounts is to physically verify each and every one in meat space, which is completely and utterly unreasonable. Even then it'd still be easy to get all of your friends who don't give a flip about K5 to get accounts for you. I don't think there is an easy solution, and even the difficult ones have undesireable drawbacks.

If people are worried about multiple account abuse, then suggest things that make multiple accounts worthless -- or make it more work to use multiple accounts. I've seen suggestions where you have to post X times and get at least a Y score before you can rate. That could work at curbing some of the abuse. I guess the question is, can you lose that privilege once it's granted to you, or is it yours forever? I see good and bad in both...


--
Without you I'm one step closer to happiness without violence.
Without you I'm one step closer to innocence without consequence.


[ Parent ]
tracking actions (3.00 / 4) (#70)
by garlic on Mon Jul 16, 2001 at 01:35:56 PM EST

This is why I said tracking their actions. There are only a couple reasons I can think of for making multiple accounts, and most of these involve modding or voting on stories.

By tracking the activity from individual ip's and studying it, you can see what activity those with multiple accounts are doing, and limit that sort of activity.

For example, you could make it so that a story submitted from one IP address cannot be voted FP by that IP address. You can also make it that each IP can vote on each story, or particular comment only once per hour.

This limits multiple accounts, but doesn't really hurt roommates

However, like I said before, I'm not familiar with the ease or difficulty of faking your IP address. Any sort of IP identification would rely on this not being easy or worthwhile to do.

HUSI challenge: post 4 troll diaries on husi without being outed as a Kuron, or having the diaries deleted or moved by admins.
[ Parent ]

Yes Yes Yes (2.60 / 10) (#47)
by PresJPolk on Sun Jul 15, 2001 at 06:39:09 PM EST

I agree, the dual-purpose moderation is a terrible mistake.

I think you're off on what can be done, though. Give two ratings per story - one for the trustworthiness of the user, and another for the value of the comment.

That way, I can give a comment a rating of a 2, or a 1, without fear of hurting that user's use of the site. There's your solution to inflation.



The answer: Correlated ratings (4.23 / 21) (#48)
by tmoertel on Sun Jul 15, 2001 at 10:29:42 PM EST

I've mentioned before (1 and 2) that one solution to the problems cited by ucblockhead would be to use a correlated rating system instead of today's average-based system. A correlated system assigns a rating to each post based on the distillation of other users' ratings for that post as adjusted by the degree to which those users' ratings have correlated with your own ratings the past. If JoeEvilUser takes out a new account solely to rate down the posts of users he doesn't like, guess what? His ratings won't correlate strongly with normal users' ratings, and thus the ratings will largely be ignored by the K5 population. System: Heal thyself!

This rating system solves the discussed problems and provides additional benefits:

  • First, users are given incentive to rate comments. The more frequently a user rates, the more detailed his rating profile becomes and the better his correlated ratings for future comments become.
  • Second, users are given incentive to rate truthfully (and, correspondingly, they are penalized for rating dishonestly): If a user rates dishonestly, he damages his own profile and diminishes the likelihood that the rating system will serve his own information needs adequately.
  • Everybody's own personal rating system will work equally well. It doesn't matter if Joe thinks that "good" comments deserve a 4 while Jane thinks that good comments deserve only 3. The system will learn that what Joe thinks is a 4 is only a 3 when Jane is looking at it. The right thing happens.
If you think that K5's current rating system is broken, consider whether the fix is correlated ratings.

--
My blog | LectroTest

[ Disagree? Reply. ]


The whole thing's been silly all along. (3.88 / 17) (#51)
by Kasreyn on Mon Jul 16, 2001 at 12:35:58 AM EST

Now honestly.

If you see a post that's "average", what's your response? Your response is, "ho hum it's average no need to moderate." If you see a great post, your response is, "wow that's cool I'll moderate it highly." If you see a total crap post you think, "This post must be flamebait or just an idiot. I'm modding it low."

Notice the lack of something? Yes. There is no situation in which one would hand out ratings of 1 and 2!! (sigh) Look. K5 moderating is based on a false assumption: that telling people 3 is average will make average posts magically wind up being modded to 3. It doesn't work that way.

There are usually 30 to 40 posts on an article when I read it. Now, assume EVERY LAST POST is average. You have just told me it is my duty to go through, and mod EVERY DAMN ONE of them to 3. Sorry, nothing doing pal. Humans are too lazy. We mod the good ones highly, and since we were told, by the K5 mod instructions page, that 5 is what we should rate a good post, we obeyed.

A better system: Posts start at a rating of 1, like on /. You can then give them +1's or -1's, also like /. This way, the average post would be reflected by its score - it would just sit there at rating 1, and never budge upwards. Having a rating above 1 would actually MEAN something once more. Having a rating of 5 would be something great, and would make people skimming the article perk up and read it to see why.

I hope I made sense, it's late and I'm tired. Basically, the system is flawed because the average rating is (unrated) (is that 1 or 0?), but K5 *tells* us the average rating is 3. This discrepancy is all you need to understand to understand the flaw in the system.

As the saying goes (IIRC), "Never attribute to malice what can be adequately explained by incompetence."

-Kasreyn


"Extenuating circumstance to be mentioned on Judgement Day:
We never asked to be born in the first place."

R.I.P. Kurt. You will be missed.
NOOOO! (4.20 / 5) (#72)
by MrAcheson on Mon Jul 16, 2001 at 02:18:31 PM EST

Please God do not move us over to the /. rating system. The reason? Once you get above about 50 comments in an article it becomes unworkable. No comment made after that can get the needed attention to reach a level of 4 or 5. No matter how good the comment is the /. ratings system gets completely overwhelmed by noise in almost no time at all, unless its on a popular thread.

As much as the current system may have its flaws, the best solution is to use a better statistical method than the mean. I personally like the median, but thats just me. Also getting users to moderate more and labelling the comment ratings in the pull downs would solve a lot of our comment problems.

The real thing though is that we really really really need a better submission system because complete crap is being posted to the front page. I really could use an article that doesn't have the words "anus" or "asshole" in the titles.


These opinions do not represent those of the US Army, DoD, or US Government.


[ Parent ]
Democratic moderation system (3.11 / 9) (#57)
by emmanuel.charpentier on Mon Jul 16, 2001 at 03:28:08 AM EST

I can't believe we are so elitist we can't go with a democratic system. Didn't we learn in the last 200 years?

It's not difficult, everyone should be able to agree or disagree with an article or a comment, votes agreeing minus votes disagreeing divided by number of participants (total number at K5H) is the percentage of agreement. Every one can have a /personal/ filter, all the stories and comments with an agreement level below that filter are not seen.

Simple and neat, personal is even better, I can browse at the level I feel like. There is no abuse as you can simply vote up or down. Well, there is still an abuse possible, the majority can reject stories or comments. But then, anyone can start another community, no?

I have no idea how to properly handle fake accounts :-(

Those ideas are all at VeniVidiVoti, which is, as some of you know already, more of a collaborative writing system, a new form of Direct Democracy. With it you can write any sort of text, constitution, novel, newspaper, book of laws. It is very alike a weblog too.

tyranny of the majority (3.60 / 5) (#60)
by garlic on Mon Jul 16, 2001 at 11:01:23 AM EST

"I can't believe we are so elitist we can't go with a democratic system. Didn't we learn in the last 200 years? "

I have. I've learned that the majority frequently stomps on the rights of the minorities. In K5, that'd be the right to express your opinion, even if it disagrees with the majority. Those comments are the ones that tend to get modded down. Everyone ends up doing at times, even if it's only subconciously.

so then another interesting question would be in a democratic moderation system, how do you prevent the majority from hurting the minority? Without the minorities, the majority is basically boring.

HUSI challenge: post 4 troll diaries on husi without being outed as a Kuron, or having the diaries deleted or moved by admins.
[ Parent ]

Not forcing minorities to be in (3.00 / 3) (#71)
by emmanuel.charpentier on Mon Jul 16, 2001 at 01:57:25 PM EST

There are communities which admit very well contradictions, they might even encourage it, and see diversity as a richness.

Yet, if a minority thinks it is being represesed, or doesn't want to participate, it should be allowed to simply fork the community. TADA!

Forking, or separating, is forbidden in our modern democracies, but why should it be so in the virtual world???

Look at the k5h fork from the other site (fork on the notion of weblog, not on the code itself) :-)

[ Parent ]

a rational rating system for the rest of us (2.66 / 12) (#58)
by surplus value on Mon Jul 16, 2001 at 08:57:01 AM EST

Everyone should either rate 1 for 'I've made it to the end of your post' or not at all for 'life is too short'. Everyone will then be able to measure their self worth according the the number of ratings, like so:
  • A high number of ratings means good job! Very many people were either moved to register their approval for your excellent opinion or sufficiently violated to protest your consumate skill at challenging and uncovering contradiction in their world view.

  • A middling number means your ideas were very intelligent but your language lacked the usual oomph we've come to expect from you. Did you forget to use the preview button?

  • A low number of ratings means your reasoning was too intricate for someone of average intelligence to follow without giving up. You should consider lowering your expectations and writing more s l o w l y.
If at least 20 of you rate this post a 1, my efforts will have been vindicated and the hour it took to write this message will have been time well spent.

---
War Inc.: No one fucks with The Great Satan.

Hide ratings? (3.50 / 8) (#65)
by spring on Mon Jul 16, 2001 at 12:15:05 PM EST

I would love a way to hide how other people have rated articles. Some days, I get sick of the whole rating thing altogether, but my eyeballs can't stop glancing up to see what everyone else thought of whatever article I'm reading or about to read.

Why yes, I am pathetic, thanks for noticing. In deference to my weakness, however, could hiding article scores become an option?

A Flaw (3.42 / 7) (#68)
by Nitesurfer on Mon Jul 16, 2001 at 01:15:31 PM EST

" In other words, a user would only be able to give out four '1's (one from each booklet) before having to give other ratings."


This is inherently flawed because you force people to be agreeable.... If a person does not care to rate something that should be okay. However with your plan he has to give an average to something he may not feel inclined to comment on at all. This also presumes all stories are read by all K5 members. I believe people only tend to read stories they are interested in, and then rate them if the story triggers an emotional response.

Why should he be forced to rate something he does not care to rate, just so he can blast or praise something he does care about?

Maybe instead of the scale for rating from 1 to 5 you add one other response : I don't care about this topic

If this was done it could count to your 2's, 3's and 4's, allowing them to praise or blast accordingly

Just a thought

.


David Byrd

CEO --- Twenty First Century Technologies, Inc.
Home of the Nite-Surfer Illuminated Keyboard

A meta vote and an opinion vote (4.00 / 6) (#79)
by mrBlond on Mon Jul 16, 2001 at 05:58:36 PM EST

How about an *option* in scoop for 2 dropdowns:

first a vote for the message (and article!):
1 crap
2 blah
3 average
4 good
5 excellent

and then an opinion vote:
1 what are you smoking?
2 I disagree
3 kinda sorta maybe
4 I agree
5 hey, my thoughts are copyright!

That way if I admit that you wrote a well argued and thought out post, but I still strongly disagree; I'd vote: 4,1
--
Inoshiro for cabal leader.
Used elsewhere... (none / 0) (#86)
by codemonkey_uk on Wed Aug 08, 2001 at 05:12:50 AM EST

This is like the system they use on photo.net, where it works quite well. Its not perfect, but it does the job. (They have a slew of other rating issues, but not ones that I see troubling kuro5hin.)
---
Thad
"The most savage controversies are those about matters as to which there is no good evidence either way." - Bertrand Russell
[ Parent ]
Why's it broken? (3.40 / 5) (#80)
by tapir on Mon Jul 16, 2001 at 09:27:08 PM EST

Looking at the stuff that I've written, I tend to agree with the comments. I've written some stuff that's real crap, and it's gotten bad ratings (in the 1-2 range) and I've written some stuff that's pretty good and it's gotten in the (4-5 range.) I feel that it's useful feedback.

If users would rather give 1's and 5's than give 3's, so what? If somebody complains that somebody gave them a 3, then that's their problem.

I'd be more interested in the statistical behavior of average comment ratings (all ratings on a comment) than in the behavior of individual ratings -- what do we know about those statistics?

Although we could certainly use more people doing ratings, I think there are some self-correcting factors. For instance, sometimes somebody will rate something unfairly because they've got an axe to grind -- usually somebody will notice this and rate it the other way. Any schemes for changing the way people distribute their comments statistically will just worsen the problem of not enough people doing ratings...

Comment Voting Statistics.... (4.00 / 4) (#81)
by Elkor on Tue Jul 17, 2001 at 12:02:12 PM EST

Like other people have commented, I like the idea of voting with words (spam, poor, average, good, excellent), and don't like the book idea.

I would also suggest that the number of people that have voted on the article be hidden until after you have voted.

Reason? It should have no bearing on how someone is going to vote. I read comments in time order (currently), so the actual score of the post doesn't "matter" to me. But, when I go to vote and glance at what other people have voted, I can't help but do a quick mental calculation to see what my vote will make the final score. Guiltily I admit that I occasionally vote my score slightly higher to boost someone's post over the 3.5 mark (damn that number!).

Second, I like the idea of an initial value for a comment being 3 or "average" but don't think that the initial vote should have any weight. If it did, this would forever prevent people from getting a perfect "5" on a comment or a perfect "0" (though I have yet to see a perfect zero) In any case, just an idea.

Regards,
Elkor

"I won't tell you how to love God if you don't tell me how to love myself."
-Margo Eve
See for your self how it fails (2.50 / 2) (#84)
by urgan on Wed Jul 18, 2001 at 07:51:18 PM EST

Look at the story about XP activation.
http://www.kuro5hin.org/story/2001/7/18/13432/2251
Scroll down, blah, blah, same old, same old, and this guy, which is obviously a stranger like, says the obvious "it was already broken" and provides a link.
http://www.kuro5hin.org/comments/2001/7/18/13432/2251/15#15
It's only useful post in the all story, it really can't be more insightful.
Guess how much he got. Yes the same I'll get. Zero.


The failings of K5 Comment Moderation | 85 comments (81 topical, 4 editorial, 0 hidden)
Display: Sort:

kuro5hin.org

[XML]
All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!