Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

Rating The Rating Scale

By farl in Meta
Wed Nov 08, 2000 at 08:46:38 PM EST
Tags: Kuro5hin.org (all tags)

In posting comments it came to me that the arbitrary rating scale that is used to moderate comments, is by its very nature, arbitrary. Since everyone has access to the 1-5 (and some access to the 0 if their mojo is high enough) rating possibilities, it would make sense to me to put some sort of vague guideline in as to how ratings work.

Please note that 1 is not "bad." It just "not as good as this other comment, which I'd rate five." All comments are assumed to contribute in some way. [as stated in the FAQ for Kuro5hin by Rusty]

This is as far as it goes for help with ratings. I personally feel that this is very vague, and in fact helps to defeat the usefulness of moderation and the rating scale.

Except for spam, all comments do add to a discussion. They might not add very well into a discussion, or contribute much of value, or even be remotely on topic, but they do enhance a discussion merely by people caring enough about a topic to take part. Evan spam can fit this profile (but very rarely does).

What interests me is how different people would rate certain comments based on their own internal scale. One of my comments which answered a question posed in the main thread, was intially rated a 2.00. Now don't think I am merely bitter about the low rating, because that aspect doesn't really matter to me in the end. What did interest me, is that when I posted a reply to this to ask "why this was done?", the response I got was: "Fact: in general, longer comments get rated higher…" My comment was a very short, very specific answer to a question posed in a comment that I was responding to. I did not need any fluff, nor a long response. The question merely wanted a list of items. So I supplied that list.

I find this trend of quantity over quality to be very disturbing if it is true, because it implies that people do not actually READ what is in a comment, but gloss over and tend to equate quantity with quality. These two concepts are VERY different as should be obvious to most people. So what I intended to do as a response to this, was to see if we could get enough feedback here that maybe the FAQ on ratings could be modified/updated to reflect a more even form of rating scales.

Here is how I rate items:
  1. 0 = Spam. (I have trusted user status most of the time)
  2. 1 = Poorly thought out. Bad/Faulty logic argument. No real relevance to topic/question. Bad structure.
  3. 2 = Does not respond to question/thread that it is relating to very well. Wanders off topic occasionally.
  4. 3 = Average answer. On topic.
  5. 4 = On topic. Good logic. Accurate/concise response to question/thread
  6. 5 = On topic. Good logic. Accurate/concise response to question. Introduction of supporting evidence and/or further information on the topic. Well written and structured.

I would like to hear about your scales and how you rate articles. If we get enough of a consensus we can draft a revision to the FAQ and approach our trusty FAQ editors and ask them if they would consider updating the FAQ with the revision. This would help create better moderated threads and comments and ultimately benefit us all. The rating scale is there for a reason and is meant to be used.



Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure


How do you rate most comments?
o 5 4%
o 4 16%
o 3 38%
o 2 8%
o 1 7%
o 0 1%
o I don't take part in the ratings game 23%

Votes: 71
Results | Other Polls

Related Links
o Kuro5hin
o mojo
o Please note that 1 is not "bad." It just "not as good as this other comment, which I'd rate five." All comments are assumed to contribute in some way.
o Rusty
o comments
o reply
o response
o Also by farl

Display: Sort:
Rating The Rating Scale | 34 comments (34 topical, editorial, 0 hidden)
I like the way it is (3.58 / 12) (#1)
by electricbarbarella on Wed Nov 08, 2000 at 06:18:30 PM EST

personally, i like the system the way it is. You can interpret it as you please (like the system you outlined in the article), or just say "eh, 1 isn't as good as 4" like i do.

lack of a rigid system is probably what keeps k5 from becoming another bastion of idiocy like slashdot.

-Andy Martin, Home of the Whopper.
Not everything is quantifiable.
i agree (4.50 / 4) (#23)
by rusty on Wed Nov 08, 2000 at 10:29:11 PM EST

That's pretty much the way I view it. Ratings are subjective; I feel like any attempt to force an objective scale would be a mistake. It would be ignored by a lot of people, and it would give the impression that there is some objective "truth" to comment ratings. The only truth is, one is low, five is high, and the overall rating at any given time represents the concensus reality about that comment.

A couple more things to note:

  • It is easy to "correct" a rating you feel is unfair. The system knows that this is possible, and counts on users doing it. If you see a comment you think is misrated, tweak it till it looks right to you.
  • The system is responsive to continued input. Any attempt to modify it to act "more like slashdot's" would be a *huge* mistake, IMO. This includes any scheme that is additive instead of averaging. I also think this includes trying to define exactly what a rating means-- "Informative" "Insightful" blah blah blah. These labels, despite the fact that they are all subjective, add an appearance of objectivity to it, which just ain't there. I don't want to pretend there's an objective basis for ratings.
  • The trust/mojo system is very resistant to the occasional bad rating. It could be more so, but there haven't been any real problems with it so far.
  • farl sees a "trend" in his one (maybe) misrated comment. I see one event at one moment in time. If anyone wants to actually see if there is such a trend, I'll provide you scads of rating data. Most reports complaining about the rating system are of this nature: "My comment got rated down. I see a trend in rating such that [X]. This is a disaster!" I have yet to see any reports that actually show a real trend, as opposed to a few outliers. That said, I'm serious, if anyone wants to do a qualitative analysis, I will dump the DB and send the relevant parts to you. I would love to have a real analysis of how rating seems to work.
As always, input is more than welcome. I do think farl's definitions of rating points are a perfectly good description. I'm not sure about the wisdom of making them "site doctrine" though. But maybe. :-)

Not the real rusty
[ Parent ]
Re: I agree (4.33 / 3) (#30)
by farl on Thu Nov 09, 2000 at 11:26:41 AM EST

I wasn't trying to make my definitions site doctrine, but maybe getting a concencus on what the typical view is (which seems to be coming out nicely in the comments and responses to this thread). More of a guideline rather than a strict doctrine.

As for a trend, I am not sure really, but what I was referring to was a comment by another user that their was such a trend towards dwindling scores. I do disagree with that and I am sure the actualy numbers on the site will bear with a trend towards 3.

What might be nice as a real small upgrade in the rating scale, is, as some people have suggested, to include the number of raters that have rated a comment. Its more information than is really needed, but I do think it is interesting information. Knowing that only 1 person rated you low on a specific comment, and that 10 rated you nicely on another, would give the poster more confidence in posting further comments and would ultimately help promote discussion.

As a kind of aside question, how about in sections that if a thread gets more than 10 or 20 comments in a set amount of time, it moves back up to the top of that topic queue. This would help keep interesting discussions alive.


[ Parent ]
2 good suggestions (4.00 / 1) (#31)
by rusty on Thu Nov 09, 2000 at 05:26:22 PM EST

to include the number of raters that have rated a comment

I'd go one further and include access to a list of who rated it what, exactly. I think this is a good idea, and transparency should help curb any potential abuse with what Sunir calls SoftSecurity.

how about in sections that if a thread gets more than 10 or 20 comments in a set amount of time, it moves back up to the top of that topic queue.

This is another good idea, and I've been pondering how to do it ever since I heard about the feature. I think I want to keep "bubble view" distinct from "chronological view"-- maybe there should be an "order by" list on story summary pages (section pages) that lets you choose which view you want. Default view could even be a user pref... That actually wouldn't be all that hard to implement.

And don't think I was criticising your article particularly. It was more reasonable than a lot of the criticism I hear. Just that there hasn't really been any objective analysis of how well rating works. I'm putting karsten on that, ex-SAS guy that he is, and we'll see if we can actually get a set of analyzed numbers to draw our conclusions from. :-)

Not the real rusty
[ Parent ]

No change (3.14 / 7) (#2)
by dreamfish on Wed Nov 08, 2000 at 06:22:48 PM EST

I'd leave it as it is. After all, as rusty pointed out, it's relative to other comments for that article. In respect of that, maybe five (or six with mojo) levels is too many - I'm not sure. It seems to work well as the general heat-to-light ratio of comments is better than elsewhere.

As to longer comments getting higher scores, people just need to be reminded why a comment should get a particular rating. It helps if they actually read the comment too :)

Ratings (3.42 / 7) (#3)
by greydmiyu on Wed Nov 08, 2000 at 06:26:43 PM EST

Personally I rate good comments, regardless of length, higher than bad comments. Good/bad is determined on whether or not it adds something to the overall coversation. I also tend to try to rate anything posted to a story I submitted just to help keep the discussion moving along.
-- Grey d'Miyu, not just another pretty color.
I'm different... (3.50 / 4) (#20)
by Pakaran on Wed Nov 08, 2000 at 08:03:15 PM EST

I usually go with 5 for trulyexceptional comments, 4 for unusally good ones, 3 for ones that were good, 2 for ones that were brief and had no good reason to be, and 1 for comments that were utterly off-topic, particularly if they were about religion, operating systems or other typical 'troll' topics.

I've never yet been trusted, so I guess I'd have to think about what to do then...

[ Parent ]

Arbitrary but equal (3.57 / 7) (#4)
by sugarman on Wed Nov 08, 2000 at 06:28:06 PM EST

In posting comments it came to me that the arbitrary rating scale that is used to moderate comments, is by its very nature, arbitrary.

Exactly. It may be an arbitrary system, but it is applied equally to anyone who posts on k5. Everone also has equality of access to the system. So it might not always work in the way you like, but you can still follow your guidelines if that is how you feel they should be applied.

The rest of us have the option to participate, or not, if we so choose. Spekaing from my own viewpoint: thats one of the things about k5 I enjoy. No one really is forcing you to think or act a certain way. It is also the reason I cringe whenever someone mentions the "representational voting" in the k5 constitution thread, or brings up the 'k5 cabal'. This is about freedom, not letting someone else decide your mind for you. <p.In the grand scheme of things, mojo don't mean nothing, so don't worry about it. Feel free to post, and rate posts by whatever criteria you think works for you.

Rating scale modifiers (3.75 / 8) (#5)
by DaveP37 on Wed Nov 08, 2000 at 06:31:20 PM EST

I tend to be pretty close to the scale farl suggested for rating, with the modification that a comment that's mis-tagged as editorial or topical gets one less than it would otherwise. It makes me grumpy when I've got the view mode set to editorial only and am still seeing topical comments (or vice-versa).

Editorial comments that explain how the article could have been better (with specific examples) generally get rated (at least) one higher than those that lack specifics.

And I may be odd, but I'm pretty sure I tend to rate shorter comments higher than longer ones, as long as they have useful content.

Seems to me ... (3.60 / 5) (#6)
by aphrael on Wed Nov 08, 2000 at 06:35:40 PM EST

That the poll results indicate things are working as designed.

Seems to me ... (4.00 / 1) (#27)
by codemonkey_uk on Thu Nov 09, 2000 at 06:20:15 AM EST

That people are fibbing in the poll.

At the time that I write this the poll results are:

5 0%
4 15%
3 41%
2 5%
1 7%

Yet every comment on this discussion is 3+, most of them between 3.5 and 4.5

Perhaps its because of the topic of conversation, which is going to make us more aware of how we moderate!

"The most savage controversies are those about matters as to which there is no good evidence either way." - Bertrand Russell
[ Parent ]
Ratings always drift down? (3.57 / 7) (#7)
by Arkady on Wed Nov 08, 2000 at 06:36:36 PM EST

Speaking of the ratings values, and going slightly off topic, but has anyone else gotten the impression that ratings tend to drop over time, but almost never go up?

Not that I'm whinging on about it or anything, and I've definitely noticed it with other people's posts, and not just my own (I want to make that clear at the outset ;-). It's just that it seems to be a general pattern, particularly for comments that get a 5; these seem to be guaranteed to drop at least to 4 over the first week.

Do you think this is just a normal function, or do you think the K5 community is expressing a general feeling that 5s are overused (_not_ an opinion I'd agree with). Is it something else; does it actually mean anything?

Just curious,

Turning and turning in the widening gyre
The falcon cannot hear the falconer;
Things fall apart; the centre cannot hold;
Mere Anarchy is loosed upon the world.

Re: Ratings drift to 3 (or so) (3.75 / 8) (#9)
by farl on Wed Nov 08, 2000 at 06:41:42 PM EST

I think it is a matter of averages.

The odds are better that most people will vote for the center, regardless of the overall quality of the comment. So over time, the drift happens.

It is more of a drift towards 3 (or whatever the average comment value is on kuro5hin), rather than a drift downwards. Maybe we just pay attention to those 5's that drift down more than we pay attention to the 1's that drift up.


[ Parent ]
attention (3.50 / 4) (#12)
by Arkady on Wed Nov 08, 2000 at 06:55:06 PM EST

I think you're probably right about drift down being more noticable than drift up. I can honestly say I haven't really looked for that.

In a community of reasonable readers (but with divergent tastes and interests, but an even distribution of difference) you would expect a drift towards the average, whatever that turns out to be.

Turning and turning in the widening gyre
The falcon cannot hear the falconer;
Things fall apart; the centre cannot hold;
Mere Anarchy is loosed upon the world.

[ Parent ]
Actually, ratings /should/ drift to average (3.88 / 9) (#11)
by ramses0 on Wed Nov 08, 2000 at 06:54:26 PM EST

It's pretty rare that comments will get a 5 in the first place. Beyond that, if 10 people "excercise their will" on a comment (that is to say: rate it), then those ten people might never agree on what comment is a five or not.

Another ~problem~ with 5's is that there is no incentive for 10 people to give a 5 comment another 5. There has been some talk about showing: (Score 4.85, 10 votes) for comments, which would be an interesting addition.

It's a problem with any rating system which takes public input- you get the average response, which usually won't trend towards 1 or 5, but instead closer to 3. For example, a comment of mine started at 1.00, and has finally settled at 2.15 ... probably where it belongs.

I actually think that the "trend towards average" phenomenon might warrant a change in mojo-

It seems like a lot of 1 ratings (not the average comment rating, but a lot of separate 1 ratings on any comment) should be a red flag. Also, a lot of 5 ratings (but not necessarily a "5" comment) should be a gold star. I'd have to think of it more, but I'm envisioning some sort of bell curve relating to mojo.

As a matter of fact, I'd be very interested to know what the "average" user's mojo is, and see if it's close to 3.00? Eh. Good stuff for scoop.kuro5hin.org. --Robert
[ rate all comments , for great justice | sell.com ]
[ Parent ]

re: drift towards average (4.00 / 5) (#17)
by interiot on Wed Nov 08, 2000 at 07:47:25 PM EST

I agree that averaging is the primary influence on a comment's score over time. But I think there's another (weak) influence that could cause comments' scores to generally drop.

On Slashdot, most people sort comments by rating. As a result, the comments with the highest rating get the most moderation done to them. On slashdot, this results in high scores being given to early posters because latecomers don't get enough exposure.

On K5, a high score is easier to get. But I'd bet that most people on K5 also sort comments by score. If moderators pay more attention to higher rated comments or even stop reading at the 2.5 mark, then a comment's rating would be more likely to be corrected if its score is above 3. So posts below 3 are less likely to get exposure and are less likely to move towards their "real" score.

Granted, the K5 crowd probably takes more effort to moderate right (but they still might unconsciously pay less attention to lower modded comments), and those who will take the time to moderate are more likely to take the time to read all the comments. But it still probably exerts a small influence. (and the hypothesis could be tested with some logging and a quick perl script)

[ Parent ]

Oh, I dunno. (3.20 / 5) (#19)
by spaceghoti on Wed Nov 08, 2000 at 08:01:26 PM EST

As a matter of personal taste, I try to view posts in chronological order sorted by thread (it's nice having a T3 that allows me to download all comments in full without having to click links) and I usually notice the ratings after I've read the post. If the post spoke to me in one way or another, I'll rate it. I have yet to see simple outright flamebait that wasn't a legitimate part of the discussion at hand, so I've never used the 0 mod, though I have used a 1 for some pretty bad posts.

I've also seen my posts modded up from 1 to about mid-2. So I agree the law of averages is really in control here.

"Humor. It is a difficult concept. It is not logical." -Saavik, ST: Wrath of Khan

[ Parent ]
Re: Oh, I dunno (3.00 / 4) (#21)
by interiot on Wed Nov 08, 2000 at 08:14:59 PM EST

It's possible you're an exception to the rule. I don't know. Is it possible to create a poll without creating a story?

I usually sort comments by rating. At the very least, if there are redundant posts, the more concise and clearer of the two is usually seen first. I could be the exception to the rule though.

Yes, I've had posts modded up from 1 too. But does that happen as often (percentage wise) as posts are marked down from 5? Only perl scripts will tell...

[ Parent ]

How people view K5 comments (3.50 / 2) (#29)
by codemonkey_uk on Thu Nov 09, 2000 at 06:59:25 AM EST

Is it possible to create a poll without creating a story?

Not nessesery, in this case, Rusty has the information, (or is it done with cookies?) he just neads to extract it and post it as a reply to this comment. (Unless its cookies...)

Rusty? Hey! Rusty! Are your listening? :)

"The most savage controversies are those about matters as to which there is no good evidence either way." - Bertrand Russell
[ Parent ]

Unrated, Then Highest (4.00 / 2) (#28)
by codemonkey_uk on Thu Nov 09, 2000 at 06:30:12 AM EST

Was the obviouse choice for me. I consider it almost a responsabilty to accuratly rate the unrated comments, because, as you said, most people, especially new users, will sort by highest rated first, and may not read all the way to the bottom. (In fact, I don't always read all the comments. Its to much.)

That said, since I became a Trusted User, I also zap down to the bottom of the story and check the moderation down their to. See if anything neads to go sub zero, or if anything got an unfairly low score.

But as I said, thats just me.

"The most savage controversies are those about matters as to which there is no good evidence either way." - Bertrand Russell
[ Parent ]
Lack of ratings... (4.00 / 4) (#22)
by ZanThrax on Wed Nov 08, 2000 at 08:48:17 PM EST

I think that many people rate to influence the total score, rather than what they feel the comment is worth. So when someone who thinks a comment deserves to be at 5 sees that it already is at 5 because only one person has rated it, they don't bother with the pulldown for that comment... Maybe the scores should be hidden for comments that a user hasn't rated yet?

Before flying off the handle over the suggestion that your a cocksucker, be sure that you do not, in fact, have a cock in your mouth.
[ Parent ]

My Interpretation (3.28 / 7) (#8)
by interiot on Wed Nov 08, 2000 at 06:37:44 PM EST

I've been thinking about this recently too.

First off, if we do have a constitution, it'll probably have quite a few of these vague-and-barely-enforced-value-statements just like most small groups' constitutions.

This is my personal interpretation of what I see others doing, so I could easily be wrong.

(1) I think one ideal of K5 is that every post should contain original thoughts, thus the MLP acronym. (1b) Exceptions are sometimes made if you post someone else's ideas that are extremely insightful and most of the community doesn't know about it (or maybe they need reminded that it applies to the current topic).

(2) If a post that don't present anything new (except as in 1b), or generally breaks some K5 etiquite rule, then it should be moderated to less than three.

(3) If a post presents something new, but it isn't that great, or if it's a MLP that's only somewhat significant, it should be moderated to three.

(4) If a post makes you think "Wow! I never thought of that", tingles run down your spine, and you reread it a couple times to make sure you won't forget it, then it should be moderated as five.

Comments like News (3.33 / 9) (#10)
by atom on Wed Nov 08, 2000 at 06:46:53 PM EST

What I never understood was why the comment rating system wasn't more like the news rating system - allow each viewer only a +1/-1 vote. I tend to vote way too many comments 1's or 5's, not because I believe they deserve those, but because their existing rating is not what I think it should be. Furthermore, many people vote low on comments with which they disagree, and high for comments with which they agree, and I don't think that's the most fair way to do it. For example, a post that just says "Bush should win" is clearly a low quality comment, but Bush supporters might tend to vote it up and Gore supporters vote it down. Along the same lines, I am quite biased when rating comments for a discussion that I'm part of. I guess there's no way to completely fix this. But if users are only allowed to +1/-1 comments then biased views won't make such an impact.

dotcomma.org - Resource for programmers
+1/0/-1 Rating (3.66 / 3) (#13)
by farl on Wed Nov 08, 2000 at 06:59:36 PM EST

Maybe something along the line of:

Total := -2 <= (Previous*No.Responses) + Rating <= 2
(so -2 and +2 would act as lower and upper bounds.)


Previous - previous rating
No.Responses - number of people who have rated this comment
Rating - what the latest person rated it

[ Parent ]
Re: Slashdot Moderation (3.25 / 4) (#14)
by interiot on Wed Nov 08, 2000 at 07:05:16 PM EST

Moderation is time-weighted, so I believe that tends to fix the fact that people moderate to get the result they want, rather than moderating what they really think the comment deserves...?

Second, one problem with Slashdot's moderation system is that the system gets less information from you. When you moderate +1, sometimes you mean to say that you think the post should be at a 5, sometimes you mean that it should be at a 3. K5's method gets more precise opinions from its users, and so the final score should be more accurate.

I propose that instead of having options 1-5, that the dropdown box reflect the final score as a result of your moderation. For instance, if one person has already moderated a 5, then your options would be 3, 3.5, 4, 4.5, and 5. That would make it fit the users' internal thought pattern. (or my internal thought pattern anyway)

[ Parent ]

Not quite... (4.00 / 4) (#18)
by atom on Wed Nov 08, 2000 at 07:57:41 PM EST

Your system isn't quite perfect, though in my opinion, better than the current system. The problem is that the first person to rate the comment has the most influence. If two people wanted to vote, one a 1 and one a 5, then whoever happened to get there first would force the other person to change his or her vote. There's also a technical problem - since the choices are determined when the page loads, then someone could vote while someone else is browsing the page. That would alter the choices, but those choices wouldn't be shown to the user who is already viewing the page. And since people don't reload often and since there's a huge K5 audience, this would be a significant problem.

I do see your point about wanting to have more of a say than "good or bad" - some grayscale is needed. So perhaps instead of +1/-1 there could be fractions. Or, as someone else pointed out, +2/+1/0/-1/-2, or something along those lines.

I know this system is a little bit more slashdot-like, but you must admit that the problems I originally pointed out with K5's system are not a problem (or much less of a problem) with slashdot, even if they have other problems. But my way allows everyone to vote, not a select few like slashdot, but doesn't rely too heavily on any one user's possibly biased opinion. I think it's more the spirit and philosophy of kuro5hin, since it's nearly exactly paralleled by the news system.
dotcomma.org - Resource for programmers
[ Parent ]
"Must be responsive to initial input" (3.50 / 2) (#24)
by ramses0 on Wed Nov 08, 2000 at 11:44:36 PM EST

"For a moderation system to be effective, it must be responsive to initial input..." (KMSelf, maybe this is it, or this one)

That's one of the best things (imho) about the system we have in place now, that maybe 2-3 people rating a single comment is almost guaranteed to get a comment very close to it's "right" level.

Consider 3 people rating, and one of them is just Evil(tm). The Evil person rates a "5" comment as a "1", the other two rate it a "5", and the total is (1+5+5=11, 11/3 == 3.67). For a "1" comment to be rated as "5", by an evil user, the math works out to: (5+1+1=7, 7/3 == 2.33).

IMHO, it's not a bad system, since most comments will never truly be "1's" or "5's". By allowing K5 users to have a full range of expression (1-->5) when rating comments, it lets the system work out pretty well.

My $0.02

[ rate all comments , for great justice | sell.com ]
[ Parent ]

Doesn't scale (3.66 / 3) (#33)
by Eimi on Sun Nov 12, 2000 at 02:15:07 PM EST

There are a lot of problems with that approach. The biggest one in my mind comes from when you have a lot of ppl rating a comment at the same time. I see this all the time on slashdot (and no, I'm not objecting to it simply because they do it, I'm objecting to it because it doesn't work for them). So someone posts a decent comment (maybe deserves +2 or +3), and 20 ppl load the pages while it's +1 or 0. "Oh cool" they say, and each rate it up. Suddenly it's sitting at +5. By making ppl give an absolute score instead of relative, we can have concurrency without problems. Additonally, if everything is +-n, there can only be a few rankings (ie, you can never get a non-integer score), which just doesn't provide as much detail. Really I can't see any benefit to it.

[ Parent ]
scales (4.50 / 2) (#34)
by atom on Tue Nov 14, 2000 at 10:37:53 PM EST

I see your point and agree with it - I've even used that argument against other people's ideas for moderation. On the other hand, isn't my proposal how the news works? People don't know what others have said before they vote, so they vote what they think (except, for comments, it would probably show the existing rating anyway - an unavoidable flaw). If you think of my proposal as voting "I like it" vs "I don't like it" rather than +1/-1, it makes more sense. People would vote "I like it" if they thought it was worthy of being read, and "I don't like it" to moderate it down. I think moderators should not base their opinions on existing opinions. Of course, this is an idealized situation - moderaters won't all listen to suggestions for how to moderate. I like it / I don't like it might or might not add or subtract 1 from the score - the algorithm could be worked out later.
dotcomma.org - Resource for programmers
[ Parent ]
Moderation is an individual thing (3.75 / 8) (#15)
by Dacta on Wed Nov 08, 2000 at 07:13:17 PM EST

Moderation is like taste - everyone has their own thing and that is the way it should be.

I don't want to have everyone moderating the same way, because that will kill the diversity of views that make the discussions on kuro5hin so strong.

For instance, I often moderate things that are technically Off-Topic up, if they are interesting and well written. Why shouldn't I? There is no reason a story that started out as a discussion about sea-urchins shouldn't contain a discussion about space exploration. K5 isn't a coprorate meeting with a fixed agenda, it is an experiment in colabarative discussion.

I tend to (a) moderate to correct what I see as faulty moderation (I usually give it a 3 or a 4), (b) mark an unmoderated comment as particully good (4), or (c) give a stupid or illogical comment a 1.

I don't usually mark down things that are factully wrong, but I try and mark up a reply correcting them.

I've never used a 0, although I have the opportunity occasionally.

There you go: that's how I moderate. I can't see why anyone should care.

The only thing I'd like to see changed is the ability to see who moderated a post and how they moderated it, similar to how story moderation works now. I think this would help deal with people who moderate something down just because they disagree with it.

M $.02 (3.66 / 9) (#16)
by spaceghoti on Wed Nov 08, 2000 at 07:19:12 PM EST

I rate when I find someone has been remarkably good at getting their point across, or if they've been remarkably bad about it. I tend to mark flames lower, but I also consider how well they're explaining why they're flaming. I also try to rate all responses to my posts, good and bad.

Overall, I'd recommend that the system be left alone (it's ambiguous, which is not a bad thing) and to not take it personally. I've been marked down to 1.00 because someone disagreed with me, apparently not because they thought I had written a bad post. *shrug* Enough people rate me high enough that it doesn't affect me overall, and I learn not to take it as a personal affront. Not worth the effort.

"Humor. It is a difficult concept. It is not logical." -Saavik, ST: Wrath of Khan

Moderation and Trust (3.66 / 6) (#25)
by PresJPolk on Thu Nov 09, 2000 at 12:03:15 AM EST

Until moderation of individual comments is unlinked from rating the trustworthiness of the poster, tweaking the commentary linked to the individal ratings is like installing a glass cockpit on the Challenger. (Shall we not upgrade our cliches with the times?)

Discussion is not all about making great speeches, and expressing breakthrough ideas. Discussion is two way, - people will make rebuttals and queries, and those will get replies in return. Don't tell me that little replies, acknowledgements, and requests for information should be made via email; that'd only be acceptable if kuro5hin were a mailing list or a newsgroup, because spreading the discussion over two different media is not practical.

Far too often, people downgrade comments expressing ideas that they disagree with. Sometimes, I'll make a comment I think is rather mundane, and probably worthy of a 3, and it'll be in the 4-5 range, just because it expresses a popular view. Other times, I'll spend 10-20 minutes writing up a view, backing it up with facts, and trying to express it very clearly, and it'll get marked down 1-3, because it's an unpopular position.

Fine, people will always mark down the things they disagree with. We can't help it; the things we disagree with will always seem to distort the truth, or to make logical fallacies. So let us accept that, and separate the trust metric, from the comment sorting metric.

How I moderate (4.00 / 3) (#26)
by codemonkey_uk on Thu Nov 09, 2000 at 05:34:29 AM EST

For top level posts my mental rating guide is as follows:

0) Spam, nonsense/first post, etc (I've only done this once, to a "First Post")
1) Badly off topic, or redundant. I don't use this very often.
2) Confusingly written, but at least shows signs of making an effort.
3) Most posts. On topic, making a point.
4) As 3, but more elequantly put, or exploring a new/alternative ideas. These comments will make me think, or bring a smile to my face.
5) As 4, but with links to relevent information, or simply amazing. This is very rare.

In addition I tend to adjust this scale, -1 for rambling nonsense, +1 for links to relevent information, -1 for bad formating (HTML posted plain text, for example), -1 for aggressive language, +1 if its also funny. (Jokes on their own don't rate that highly, but wit in context can.)

I don't rate the really long posts, and I don't usually read them. :)

I think I rate replies on a different scale. They are more likly to get a higher score by simply answering a question, or refuting a point, I like to think doing this promotes discussion.

I hope this helps.

PS - Thanks for making me think about it.
"The most savage controversies are those about matters as to which there is no good evidence either way." - Bertrand Russell
My interpretation of ratings on my comments (4.00 / 4) (#32)
by loner on Thu Nov 09, 2000 at 06:22:16 PM EST

I usually don't rate comments much, but I do pay attention to how my own comments are rated. So for your reference, here's how I interpret the scores you give to my comments:
  • 5 - Damn right! Obviously, all my comments are truly amazing, and it's good to see that K5 users have good enough taste and are intelligent enough to notice this.
  • 4-5 - Well I suppose one can't expect every K5 user to be so intelligent and have a good enough taste to see that this comment should be scored with nothing but a "5".
  • 3-4 - Hmm, the weather must be particularly bad all over the world today, making people grumpy. Why else would K5 users score so low such a brilliant 5-star comment as I have made?!
  • 2-3 - Are the schools closed today?! There seems to be an awful lot of kids messing with K5's rating system, and scoring my comment so badly. I mean, c'mon any mature person will see that my comment deserves nothing but a 5!
  • 1-2 - That's it! screw the lot of you! I'm going back to slashdot... ... OK, maybe not yet, I'll give you one more chance.
Wait, wait! Before you click on the RateAll button, consider this. Also keep in mind that I just got my trusted user status back. So please be gentle. Please! :)

Seriously though, I had similar reactions when my comments started getting scored lower than 2. I do put a lot of work and thought in all my comments, and I'm surprised by some of the scores. I'm even guilty of starting a few threads on changes to the scoring system. But now I think the system is fine the way it is.

Right now, the only things the scores affect are: one's mojo, one's ego, and the sorting of the comments. The mojo means little, nobody posts spam to rate down to 0 anyway, and one's ego means even less. All we're left is the sorting thing and right now the quantity of comments is low enough that sorting is not that big a deal.

Maybe as time goes on, the number of comments will be large enough to make sorting by score a necessity. But by then, people will naturally start scoring by "you gotta read this" and "don't waste your time with this one." For now, as long as everybody scores all my comments as 5, I don't see any reason to change.

Rating The Rating Scale | 34 comments (34 topical, 0 editorial, 0 hidden)
Display: Sort:


All trademarks and copyrights on this page are owned by their respective companies. The Rest © 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!