Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

[P]
Comment ratings and bias

By valarauko in Meta
Sun Feb 24, 2002 at 02:25:10 PM EST
Tags: Kuro5hin.org (all tags)
Kuro5hin.org

I find that when I read a comment, I already have a built-in prejudice for or against it based on the rating other users have given it. I think it would be a great feature to be able to hide comment ratings until you've already rated it yourself.


This prejudice carries over when I rate the comment, leading to follow-the-leader behavior where the first person to rate sets the trend for the next 10 (I am assuming that other people are swayable as I am -- this may not be the case).

I appreciate the fact that a story submission's score is not shown until one votes on it. Maybe comment ratings don't need this kind of assurance of impartiality, but I think the option would help things.

Sponsors

Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure

Login

Poll
Hide comment ratings?
o Yes 13%
o No 23%
o I want the option 53%
o I think it's a waste of brain-width 8%

Votes: 117
Results | Other Polls

Related Links
o Also by valarauko


Display: Sort:
Comment ratings and bias | 31 comments (29 topical, 2 editorial, 0 hidden)
+1 This has potential (4.44 / 9) (#2)
by cleo on Sat Feb 23, 2002 at 09:56:40 PM EST

I'd like the option to choose whether I see comment ratings. When I have 10 minutes to browse new comments before I go to bed, I like having the guideance of seeing what other people have found worthwhile.

When I'm rating comments, though, I agree that previous ratings prejudice me (either to follow suit or to violently disagree with the existing rating). I'd be much more impartial if I could hide ratings until after I've made my own decisions.

You are a helper, not a judge (4.42 / 7) (#9)
by BlowCat on Sat Feb 23, 2002 at 11:47:26 PM EST

You are not a judge when you are rating comments. You are a helper. You are helping those who have no time to read all comments, just like you before you go to bed.

If you see an interesting comment that was wrongly (in your opinion) rated, you can use 1 and 5 to shift the rating as much as possible.

I have no problems with you doing so as long as you honestly believe that I should (or should not) see that comment if I'm only looking for the best comments.

[ Parent ]

I agree. (4.20 / 5) (#3)
by valeko on Sat Feb 23, 2002 at 10:08:53 PM EST

Another thing that discourages you from rating on comments based on other ratings is the perception that you will have no bearing on its rating when the average is entirely in the opposite direction. Mind, I don't rate comments in order to swing them in various directions, but nevertheless if you find a comment to be worthy of a 4 or 5 and it's at (1.00 / 3) already, that can have an effect on your willingness to rate it.


"Hey, what's sanity got going for it anyways?" -- infinitera, on matters of the heart

Yeah, but usually it's a GOOD influence (4.42 / 7) (#12)
by UncleMikey on Sun Feb 24, 2002 at 12:24:29 AM EST

Well, for me, anyway. Being something of a contrarian, if I see a comment I think has merit that's gotten tweaked downward, I vote my conscience anyway. I'm actually more likely to actually rate a comment at all if I think it's been rated down unfairly. The reverse is not necessarily true, however -- I'm not so much likely to bother rate something low if many others have rated it high.
--
[ Uncle Mikey | Radio Free Tomorrow ]
[ Parent ]
It doesn't matter, at least, most of the time (4.25 / 4) (#16)
by whatwasthatagain on Sun Feb 24, 2002 at 03:32:12 AM EST

If I think a comment has been rated unfairly (both low rating for good comments and high rating for bad comments), I vote to correct it. Well, I'll admit that I do a little "steering" - but only when I think the comment has been rated unfairly.

It's for the marginal differences between the rating of a comment and my perceived rating that I go with the crowd. These don't affect the rating of the comment, but add to the numbers. Why then, do I do it at all? Because it adds to the numbers, and makes vote steering by others more difficult. I thus increase the chances of the comment staying at my perceived rating.
--

With profound apologies to whomsoever this sig originally belonged.
[ Parent ]

Following the lead is not always bad. (4.66 / 9) (#4)
by BlowCat on Sat Feb 23, 2002 at 10:10:23 PM EST

It happened to me many times that I voted -1 on a story to find that it had about 80 points. I felt sorry that I voted against something that so many people want to be published.

Often the reason why I vote -1 is because I see (usually high rated) comments asking to rewrite the story. I vote -1 in hope that the author will indeed rewrite the story, but then I find out that most users want to see the story as is. Since I'm interested in the topic, I feel that my vote is in fact delaying the full-blown discussion.

Maybe the voting should be two-dimensional: interest and quality. Then I could vote +1 interesting and -1 rewrite in the same time.

Example: I really wanted to see an obituary for Daniel Pearl, but I felt that he deserves more that just an MLP story.

I think that's your mistake (4.66 / 3) (#6)
by Gutza on Sat Feb 23, 2002 at 10:39:39 PM EST

Two of the three stories I posted on K5 bounced once before being accepted - mostly due to my lack of talent in writing, I must admit. But when I saw that the score was hanging at about +15 and already 50-60 people voted on it, I just posted an editorial "PLEASE VOTE THIS DOWN!" comment so I could get it over with. So your vote should not be biased by the reason you describe - unless the author posts an obvious recommendation to vote it down, you should really vote the way you see fit...

Just my 2c.

Who's your vendor, who's your vendor? — Scott Adams
time is K5
[ Parent ]

How about this idea? (4.80 / 5) (#14)
by joecool12321 on Sun Feb 24, 2002 at 01:11:29 AM EST

What if the author of a story could 'kill' their story by voting -100? It'd be just the same as if 100 people voted -1 (as in, what happens to the story in scoop code). This allows you to kill a story you wrote, in order have a better version posted, rather than a medocre one that gets posted at the 350 vote threshold.

It seems like you'd needs some other restriction: if there are over 30 comments you cannot kill it, for example. And perhaps it should only be -80 points, in case someone decides (for whatever reason) to attempt to kill a story in the 60's or 70's.

Well, whaddya think? Does this sound like a good idea?

--Joey

[ Parent ]
Danny (5.00 / 3) (#13)
by UncleMikey on Sun Feb 24, 2002 at 12:27:14 AM EST

Example: I really wanted to see an obituary for Daniel Pearl, but I felt that he deserves more that just an MLP story.

I agree, I wish I'd had the time to write one. As it was, few of the Editorial comments on that one were 'please rewrite' or 'please flesh out' but rather 'why is this here?'. *Shrug*

Editorial comments are key, particularly for down-votes. If you vote something down for reasons that haven't already been stated, please speak up!
--
[ Uncle Mikey | Radio Free Tomorrow ]
[ Parent ]

What's the point of voting then? (4.75 / 4) (#20)
by ahsyed on Sun Feb 24, 2002 at 06:31:42 AM EST

You're supposed to vote on a story based on it's strengths and weaknesses. Treat the story as a whole. And then when you do vote, you shouldn't care about what other people thought of it. Who cares? It's your vote.

Meaning when you vote for a President, you don't say "Hmm, I'll vote for him because I like his stance on environment but don't like his stance on abortion". You vote for him as a whole. You either like the candidate or you don't. And let's say your candidate loses, you don't say "Damn, I should of voted for the other guy because he was clearly better".

That's how it should be for a story.

[ Parent ]
Defeats the point (4.73 / 23) (#5)
by Eloquence on Sat Feb 23, 2002 at 10:32:35 PM EST

Comment ratings are not meant to be a gratification mechanism, although they are used in the diaries as such. They should help you find high quality comments, e.g. by sorting highest first. Hiding them defeats their main point. This is different from story votes, where a vote decides on whether the story gets posted or not. The comment rating (with the exception of 0-ratings) is not decisive like a story vote.

It would be a good idea to not show (and evaluate) comment ratings until there have been at least n ratings, though, where n probably should be something like 4 or 5. This both reduces the bias problem and makes ratings more useful as a quality selection mechanism.
--
Copyright law is bad: infoAnarchy Pleasure is good: Origins of Violence
spread the word!

I agree on both counts (4.57 / 7) (#7)
by Delirium on Sat Feb 23, 2002 at 11:01:35 PM EST

If you can only see comment ratings for comments you've voted on, there's no reason to have comment ratings at all.

But it would be nice to perhaps require a certain number of them to keep one user from skewing things.

[ Parent ]

Actually, comment-moderation can hide posts (4.20 / 5) (#11)
by UncleMikey on Sun Feb 24, 2002 at 12:21:53 AM EST

Users who have enough 'mojo' to attain Trusted status can rate comments down to '0'. If enough 0 votes drag the comment's rating down below 1, the comment is hidden from most users. Other Trusted Users can see hidden comments, either within a story, or in a separate section of the system, so that they can police each other and reverse decisions that seem marginal (or reinforce those that don't).

From my own experience, most comments that get hidden this way are obvious trolls, flames, or spam. There's remarkably little abuse (e.g. people rating comments to 0 solely on the basis of who posted them).
--
[ Uncle Mikey | Radio Free Tomorrow ]
[ Parent ]

Uh huh, and... (4.00 / 5) (#15)
by J'raxis on Sun Feb 24, 2002 at 02:12:08 AM EST

I believe that’s what he meant by, “The comment rating (with the exception of 0-ratings) is not decisive like a story vote.”

— The Raxis

[ J’raxis·Com | Liberty in your lifetime ]
[ Parent ]

You beat me (4.25 / 4) (#18)
by valarauko on Sun Feb 24, 2002 at 04:47:21 AM EST

When I read I read K5 it's usually in 2 modes: information-gathering and judgment (for lack of a better term) (see cleo's comment). Ratings are useful for browsing, and not so useful for doing your own rating.

"It would be a good idea to not show (and evaluate) comment ratings until there have been at least n ratings"
This would produce the same result as my suggestion but with less overhead. You could rank unrated first if you're in the mood to evaluate others' comments, or highest first if you just want to see the best stuff out there.

Now submit that suggestion! +1 from me



[ Parent ]
"being able to" != "change to" (4.85 / 7) (#19)
by ahsyed on Sun Feb 24, 2002 at 06:23:23 AM EST

All that the author is asking is to have an option to disable comments. He's not saying that everyone should stop viewing the ratings, just that he doesn't want to. He doesn't want to see if something is the cream of the crop, just the crop. What's the big deal with that?

I don't know how hard it would be to code, but I would like to see it as well.

[ Parent ]
How about option to hide (4.40 / 5) (#21)
by etherdeath on Sun Feb 24, 2002 at 12:40:10 PM EST

This means you can hide the comment rating yourself, vote, then unset the hide option, reload and then you can see them.

Hiding comments is a tool for the person moderating who can't trust themselves to rate without bias from seeing other ratings; I'd probably use the feature myself.

[ Parent ]
So... (2.94 / 18) (#8)
by Kaki Nix Sain on Sat Feb 23, 2002 at 11:27:09 PM EST

... you have some trouble thinking for yourself and want a quick technical fix to protect you from too much information? Is it really too hard for you to just not look at that number, or to just ignore it as you do your oh so careful deliberations? Come on. Why don't you try addressing the real issue?



What do you mean? (3.75 / 8) (#17)
by valarauko on Sun Feb 24, 2002 at 04:28:53 AM EST

"Is it really too hard for you to just not look at that number, or to just ignore it as you do your oh so careful deliberations?"
Don't think about pink elephants.

"... you have some trouble thinking for yourself and want a quick technical fix to protect you from too much information?"
Forgive me for being one of those hu-mans who can't always separate emotion from reason.

The real issue is that many people, including myself, find it difficult to not be swayed by others' opinions, especially when those people are [generally] knowledgable and thoughtful, as the Kuro5hin readers are. Story submission scores are hidden for precisely this reason. Having the _option_ of hiding comment ratings cannot harm things, as long as the default remains to show them. The normal function of ratings, as a way of filtering information to save time, remains.



[ Parent ]
+1section (4.16 / 6) (#22)
by Cal Jayson on Sun Feb 24, 2002 at 04:55:34 PM EST

because everybody did
--
kx.com: 2.5 billion trades
select max price from trade takes 1 second
k5 comments (3.60 / 5) (#23)
by bugmaster on Sun Feb 24, 2002 at 06:36:47 PM EST

You know, I just realized that I don't even pay attention to the comment ratings anymore. Usually there aren't enough comments for ratings to matter; and the comments that are there are usually fairly mature. The troll density seems to be much lower here than it is at The Other Site (tm). I wonder why that is ?
>|<*:=
A lot fewer people (3.00 / 1) (#25)
by wiredog on Sun Feb 24, 2002 at 09:48:25 PM EST

I remember when slashdot was fairly tolerable. I dread what k5 will be like after it'd got 500,000 accounts or so.

Peoples Front To Reunite Gondwanaland: "Stop the Laurasian Separatist Movement!"
[ Parent ]
Slashdot utopia (4.00 / 1) (#28)
by jonathan_ingram on Mon Feb 25, 2002 at 08:17:40 AM EST

I dread what k5 will be like after it'd got 500,000 accounts or so.

True, but Slashdot's moderation system was falling apart long before it had 500,000 accounts. I'm watching Kuro5hin with interest to see how it (and the moderation system it uses) scales as the userbase increases.

To report on my own personal experience, I find the ability to rate any post I want very liberating, and a little addictive. I do worry a little about the fact that it's using an absolute scale from 1 to 5. It would only take a relatively small group of people, all moderating every single comment as 1, to hurt a moderation system without specific safeguards.
-- Jon
[ Parent ]

That has happened (4.00 / 1) (#29)
by wiredog on Mon Feb 25, 2002 at 08:39:34 AM EST

It's why several of us tend to get torqued off whenever indymedia stories show up. The im guys do that sort of thing all the time.

Peoples Front To Reunite Gondwanaland: "Stop the Laurasian Separatist Movement!"
[ Parent ]
Yup (3.00 / 1) (#31)
by cafeman on Mon Feb 25, 2002 at 09:29:34 PM EST

I got whacked by a few people b/c I had the audacity to suggest that substantially fewer protesters showed up to a demonstration than were reported on Indymedia. Forget the fact that I was there, and the person reporting wasn't. That and the mentality that if you're not part of the "solution", you're part of the problem. Pain in the butt, but what do you do?

If you make a game out of trying to get trusted user status, it would hurt, but if you come to the site for the discussion, it's just annoying. Like a fly buzzing around your head. Except the fly is more annoying. Or something like that ....



--------------------
"No Silicon heaven? But where would all the calculators go?"


[ Parent ]
Comments are sorted by rating (3.66 / 3) (#24)
by Lionfire on Sun Feb 24, 2002 at 09:41:34 PM EST

Although hiding the exact comment ratings might reduce "me too" moderation, it is important to remember that the comments are sorted. This is, for those that have forgotten, the reason we have comment moderation at all.

To completely hide the comment ratings, we would need to display comments in an unsorted manner, thereby defeating the purpose of the existence.



[ blog | cute ]
It's a preference (3.00 / 1) (#26)
by valarauko on Mon Feb 25, 2002 at 02:53:09 AM EST

You can change the sorting options... to unrated first, or lowest first, or unsorted. Scoop is pretty spiffy.

[ Parent ]
It's a preference now, but... (3.00 / 1) (#27)
by Lionfire on Mon Feb 25, 2002 at 03:18:20 AM EST

The point isn't that scoop can't show things in an unsorted manner, but that this couldn't be a preference for blind ratings to work properly. If the user has the option to sort the comments, the moderation can no longer be considered blind.



[ blog | cute ]
[ Parent ]
Valuable time saved by comment ratings (4.00 / 1) (#30)
by sean23007 on Mon Feb 25, 2002 at 05:34:07 PM EST

What about those of us who lack infinite time and want to read only the comments that are most likely to be worth the time we consider precious? If there were no ratings for comments, all comments would be given equal berth, and I would have to sort for myself the good from the bad on the fly. Sometimes this may not be too hard, as in the case of trolling and spamming and first post, etc... but what about comments that are long and professional-looking but end up being just a worthless rant? I don't want to have to read such a thing, and I want other people to be able to make that judgement for me, so I can read only the comments I feel might be most worthwhile. If I am really interested in the topic/story, I will delve deeper into the page, and read the comments that are lower rated, until I get down to the downright worthless ones.

To summarize: ratings == good.

Lack of eloquence does not denote lack of intelligence, though they often coincide.
Comment ratings and bias | 31 comments (29 topical, 2 editorial, 0 hidden)
Display: Sort:

kuro5hin.org

[XML]
All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!