Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

[P]
How much harm from Exposed Ratings?

By Parity in Meta
Wed Jan 10, 2001 at 07:13:33 PM EST
Tags: Kuro5hin.org (all tags)
Kuro5hin.org

I believe enough time has passed since its introduction, that we should discuss the effects of the exposed (or, in Rusty's words, 'accountable') rating system.


I think the exposed ratings are more harmful than not, and that secretly administered unfair ratings may have been ended, that the current situation has lead to a kind of 'rating inflation' where any rating below a 4 must be a personal criticism or attack of some kind. Today was the first time I saw someone actually criticize the rating s/he was given, here if you must know, but chances are that there have been others that I didn't see, and I don't mean to single out that user in particular.

I did, and will continue to, rate 'meta-debate' like that to a 1. It contributes absolutely nothing to the conversation, tangenting it off into an interpersonal argument of no interest to anyone except the rater and the ratee.

Aside from the potential for ratings themselves becoming the topics of arguments, not to say flamewars, the threat of that possibility seems to mean that people now rate '5' for good, '4' for 'fair' and '3' for poor, and anything less is reserved for personal animosity ... or used only by those few who, like me, continue to rate 5 for excellent, 4 for very good, 3 for pretty good, 2 for a reasonable but boring comment (the predictable statements that have been made thousands of times about MS or Linux, for example, or simple 'me too' or 'no way' comments), and 1 for 'inappropriate'.

I know for certain that my personal 'good' comments have suddenly started getting astronomical scores (perfect 5 that nobody wants to downgrade, high 4s... ) and that my less good comments, which priorly would have rated say, a 3.2, are no left conspicuously 'Unrated' ... presumably because nobody thinks they're worth giving a 4 or 5, and nobody wants to 'insult' me with anything lower. (The exception, if you check up on me, at 1.5 was a metacomment that fluffy grue didn't like... I don't think that exception means the trend isn't there.)

All of this together makes me worry about three things:
One, that the reward/punishment flavor that ratings seem to be acquiring will increase the feedback effect of the community, encouraging opinion-consensus and groupthink.
Two, that an increasing amount of time will be spent on think about and talking about the ratings themselves instead of letting them just be a mechanism by which posts are ordered.
Three, and perhaps it should have been one being the most clear and present concern, that ratings will become increasingly meaningless as more and more people use only 4 and 5 votes occassionally daring a 3.

Okay, so that's what I'm seeing. Is everyone else seeing the same thing? Do you think the new system is better? Is Rusty even willing to think about undoing this and try something like, say, removing diary ratings from Mojo calculations? (Aging takes care of post-rating dead conversations, leaving diaries as the only hole for 'undetected' ratings/mojo attacks.) Has the fear of your rating being observed affected how -you- rate? Or how your comments are rated?

Sponsors

Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure

Login

Poll
Exposed/accountable ratings are:
o Better than Hidden Ratings 58%
o Worse than Hidden Ratings 12%
o Neither better nor worse than Hidden Ratings 11%
o Not worth caring about 17%

Votes: 79
Results | Other Polls

Related Links
o Kuro5hin
o introducti on
o here
o Also by Parity


Display: Sort:
How much harm from Exposed Ratings? | 77 comments (77 topical, editorial, 0 hidden)
Hmm... (3.72 / 11) (#1)
by 11223 on Wed Jan 10, 2001 at 04:57:08 PM EST

I'd have to say they were a good thing. If you look at the ratings on the comment I complained about, you'll see that most of the other ratings were 4's. Out there, all by its lonesome self, was a 2. Now, that causes me to look at the user doing the rating, and to question it because I know that Gandalf is a big BeOS guy, and was wondering if he was voting politically.

In another example, have a look at this comment. Quite a few 3's, 4's, 5's.... and then a 1. Now, this would seem a bit out of the ordinary... until you look at the user, 'untrusted user', who happens to be a big-time spammer.

On the whole, there isn't really harm. In fact, I think the extra accountability from open ratings will expose those who are voting politically or as spam instead of by comment worthfulness, and help us find the stalkers.

--
The dead hand of Asimov's mass psychology wins every time.

Possibly... (3.75 / 4) (#2)
by Parity on Wed Jan 10, 2001 at 05:00:23 PM EST

It might be the some people rate 'politically', but it's also possible that they are 'over-rating' in order to have a bigger impact on the total of a score they disagree with. (I try not to do that, but I can understand the temptation, especially when you think a rating is egregiously wrong.)

In any case, exposed ratings encourage people to 'second-guess' the reasing of the rater, which I think is an unhealthy pasttime in and of itself.

Parity Odd


[ Parent ]
Not necessarily! (4.00 / 2) (#5)
by 11223 on Wed Jan 10, 2001 at 05:04:50 PM EST

Remember, K5 is a growing entity, not a static one, and as such it's good to call these things to the community's attention every once in a while so people understand why they shouldn't be voting politically. Besides, spam voting should always be exposed.

Here's a suggestion: let only the trusted users see the ratings, so they can identify the spammers.

--
The dead hand of Asimov's mass psychology wins every time.
[ Parent ]

Stalkers? (3.00 / 2) (#12)
by Wicket on Wed Jan 10, 2001 at 05:35:37 PM EST

Granted, I am pretty new to the site, but what do you mean by finding stalkers? What goes on at this site that I don't know about? Hehehehe :)
intune.org - music discussion for the soul...
[ Parent ]
Stalkers = (3.00 / 1) (#15)
by itsbruce on Wed Jan 10, 2001 at 05:55:00 PM EST

People who track all the comments of someone they have taken against and mark them down.


--

It is impolite to tell a man who is carrying you on his shoulders that his head smells.
[ Parent ]
I changed my mind half way through this (4.00 / 5) (#21)
by itsbruce on Wed Jan 10, 2001 at 06:23:54 PM EST

In another example, have a look at this comment. Quite a few 3's, 4's, 5's.... and then a 1. Now, this would seem a bit out of the ordinary... until you look at the user, 'untrusted user', who happens to be a big-time spammer.

Heh. I've noticed him doing that to me and others.

On the whole, there isn't really harm. In fact, I think the extra accountability from open ratings will expose those who are voting politically or as spam instead of by comment worthfulness, and help us find the stalkers.

Whoa! Now you're getting scary. That sounds very inquisatorial. Are you proposing that we analyse all the ratings and then label people by the way they vote? If that happens, I'm out of here. In fact, now that I think of it, that could be done now, so someone is likely to do it and publish it - how long before it's a de facto part of kuro5hin whether or not Rusty accepts it? That one comment from you has <honestly> just converted me to Parity's point of view.

I dropped out of kuro5hin a couple of months ago to concentrate on a spiralling workload. kuro5hin was doing a find job of being a community then, without exposed ratings. It still seems fine now - but will that survive ratings-analysis witch-hunts?

A while after I joined kuro5hin, I realised that not only was this a discussion forum with a better spirit than I had found anywhere else, but that I was behaving better than I do in other, harsher, forums. That was without any exposed ratings to hold me to account. It was partly the structure, partly the input of the community. I trusted the way I was rated without knowing who rated me up or down. I trusted the community to average things out and so cancel out the effects of stalkers.

Who cares if there are stalkers and trolls? I don't want to hunt anyone down. The community is stronger than they are - unless paranoia weakens it. Frankly, I'd think that exposed ratings are more likey to encourage stalking than to quell it.


--

It is impolite to tell a man who is carrying you on his shoulders that his head smells.
[ Parent ]
I like them. (3.36 / 11) (#3)
by error 404 on Wed Jan 10, 2001 at 05:02:55 PM EST

I've used them as a kind of network: people care enough to rate my comments, good or bad, I look at their diaries and comments. See where they are coming from. Understand my fellow online text emitter better. Get a clue as to what it was they found worth rating up or down.
..................................
Electrical banana is bound to be the very next phase
- Donovan

Wow, look at all the ratings! (4.00 / 2) (#49)
by error 404 on Thu Jan 11, 2001 at 11:42:19 AM EST

I don't think I've ever had ten (the number as of this posting) ratings on a comment before.

Could it be that some of those people are intentionaly using the rating system for networking? Cool! Both because networking is a good community-building thing and because more ratings mean more statisticaly meaningfull ratings.

I'm still not using the ratings to sort comments, but I probably will as the site grows. And the ratings are interesting.
..................................
Electrical banana is bound to be the very next phase
- Donovan

[ Parent ]

Who's that sensitive? (3.18 / 11) (#4)
by sl4ck0ff on Wed Jan 10, 2001 at 05:03:26 PM EST

OK, probably more people and even me more then I care to admit, but most people (as far as I know) don't care too much about the ratings. It should be a compass to tell you whether you're using your "interesting" ability for people to move their mouse (or Lynx users, arrow keys) to rate...Which may or may not mean they truly enjoyed your comment...I wouldn't take it too seriously if I were you.
/me has returned to slacking
I personally haven't looked (much) (3.80 / 10) (#6)
by rednecktek on Wed Jan 10, 2001 at 05:06:51 PM EST

I peeked at a comment I made a few minutes ago. One user, rated 4. I thought it deserved it, I stayed on topic, and tried to add to the discussion. Probably be around a 2 by tommorrow and I'll forget to look. More importantly, I won't care.

My point is does anyone really "study" their ratings. Do you care if XYZZY user (there isn't one is there?) rates everything you post at a 1? Do you know the users who don't like you, and worry about what they'll rate your comment? I don't. I'm not here for the ratings.

If I look at a bad one and it makes me work harder on the next post, that's a Good Thing(tm).

Just remember, if the world didn't suck, we'd all fall off.

Well... (3.00 / 3) (#7)
by Parity on Wed Jan 10, 2001 at 05:13:31 PM EST

I don't really care how my -own- comments are rated; my concern is that in a busy article (too much discussion to read every post) that the ratings should sort the articles into some semblence of importance or insightfullness. I think they used to do that. I don't think they do anymore.

Parity None


[ Parent ]
We don't live in an ideal world. (4.06 / 15) (#8)
by Inoshiro on Wed Jan 10, 2001 at 05:19:31 PM EST

If we lived in an ideal world, people would accept peer reviewed ratings, and not have a big problem if one or two weirdoes rated them really oddly. But people had a problem with it, or suspected people were 'out to get them' (sigh). So Rusty came up with the idea of showing who's rated what.

In an ideal world, people would've gone, "ahh -- that's the crazy one I'll ignore." And been done with it. Unfortunately it's led (as you said) to inflation of comment ratings, as well as pointless 'dick wars' as people continue to dither over numbers on a website which have no meaning beyond the moment.

I suggest people read the comment rating guide so that we can have a somewhat return to normalicy without hacking any other kludges onto the rating system. 'Normal' comments which may contribute a little should be 2. Unless your comment is below 2, I'd not take it so personally. And even if it is rated below 2, it doesn't mean you're going to die, or that everyone hates you.



--
[ イノシロ ]
Comment Rating Guide (3.77 / 9) (#16)
by Morn on Wed Jan 10, 2001 at 05:55:42 PM EST

Perhaps a link to the comment rating guide, making the 'rating and replying line' appear like this: [ Reply to This | [menu][Rate All] | Rating Guide ] Would be appropriate? I certainly can't see it generating any /harm/ and anything that encourages responsible rating is good in my book.

[ Parent ]
That matches what I rate. (3.00 / 2) (#58)
by static on Thu Jan 11, 2001 at 05:40:33 PM EST

I'm extremely reluctant to give a post a 5. It's the old dilemma: if I rate a really good post at 5, how do I know another one a little further down the page wouldn't deserve a 6?

As it happens, I don't rate much at all. This article may change that for a few days. :-) Normally, I will rate a post as a 3 if I think it simply needs more votes.

Wade, who doesn't keep track of who rates his own posts.

[ Parent ]

Don't fear the Mojo... (3.80 / 10) (#9)
by Electric Angst on Wed Jan 10, 2001 at 05:21:00 PM EST

I prefer the 'exposed/accountable' ratings. it allows a chance to be political with your vote, stating your position without having to reply.

For example, I can see a comment that is based around a Postman 'Amusing ourselves to Death' or Huxley 'Brave New World' argument, and since I consider these arguments to have the same merit as a conspiracy theorist trying to say we're living in a 1984-like world, I will rate it lower than a comment with similar amount of effort put into it that bases its logic on firmer foundation. Since the ratings are exposed, my position is publicly known, and I don't have to reply to every single one of these posts I see, which I have neither the time nor the desire to accompolish. The people who see my low vote may wonder why I voted in such a way, and then search my diary or comments, which might turn up an argument against the type of logic I voted down. Thusly, I am able to make a statement in a much stronger way than if I had simply rated the comment low without any sort of accountability.

In short, I take responsibility for my actions, and have no qualms about how I vote. So, I have no problem with the exposed rating system, and can't really see how anyone with similar self-accountability would object to this method.

Yes, this does ramble, but it really sucks to try and edit while in a text-box, so this is what I will post...
--
"Hell, at least [Mailbox Pipebombing suspect Lucas Helder's] argument makes sense, which is more than I can say for the vast majority of people." - trhurler
Ours is not to reason why... (3.00 / 3) (#10)
by Parity on Wed Jan 10, 2001 at 05:27:53 PM EST

... or maybe it is; in any case, I don't know -why- people are afraid to rate honestly... well, I do, actually, they don't want to be criticized for being too critical or 'hurt' by being vengeance-rated or whatever... however, whether you or I or anyone else can see the reasons, it still looks like many people are changing the way they rate. Just because it doesn't make logical sense doesn't mean it doesn't make sociological sense; people, after all, are not creatures of pure logic.

Parity None


[ Parent ]
You've missed the point (3.20 / 5) (#11)
by GreenCrackBaby on Wed Jan 10, 2001 at 05:34:45 PM EST

Clearly, you've missed the point of rating here. Rating isn't meant to be a shortcut for you to voice your opinion about someone else's post. Rating is meant to identify statements that are very interesting, shed light on the topic, are thought provoking, etc -- whether or not you agree with what is being said.

If someone makes a good argument supporting M$'s buisiness practices, even though I'll probably disagree, I'll rate it highly since it's a good statement.

[ Parent ]

What makes a 'good' comment? (4.00 / 4) (#18)
by Electric Angst on Wed Jan 10, 2001 at 06:04:13 PM EST

(Okay, I tried posting this once, and something went wrong... Let's go at it again...)

This all depends on what you consider to be a 'good' comment. I consider one of the most important traits of being a good comment being well thought-out. A comment based on an alagory from fiction or the philosophy of a single work are hardly as 'good' as an equally well-written comment that has a more diverse base upon which to build its logic.

This, of course, ties in to a major problem with web-logs, the amount of argument and the lack of backup resources. A friend of mine once commented that it seems like much of what is said here seems to be written like a research paper, and yet there is no research. I have to leave now, so unfortunantly I will not be able to delve deeper into this topic, but I stand by my ratings, and always will.
--
"Hell, at least [Mailbox Pipebombing suspect Lucas Helder's] argument makes sense, which is more than I can say for the vast majority of people." - trhurler
[ Parent ]
Well, if all else fails read the FAQ I guess (3.00 / 1) (#50)
by GreenCrackBaby on Thu Jan 11, 2001 at 11:58:22 AM EST

http://www.kuro5hin.org/?op=special;page=comments#rating

Rather than having me try and give my interpretation of how comments should be rated, and debate with you about the differences between my method and yours, I guess we could always read the FAQ. :)

The gem that I was trying to get at is this:

Likewise, if the comment is clearly directed at a certain group of people, or a certain "type" that you do not belong to, do not automatically rate it down.

[ Parent ]

My only gripe is... (4.10 / 10) (#13)
by elenchos on Wed Jan 10, 2001 at 05:35:43 PM EST

...when I move back and forth from trusted to untrusted. It is annoying. I'd like it to stay one or the other for longer periods of time just to not have to be distracted by the issue. It's like an automaic transmission in a car that can't decide what gear it wants to be in, so keeps hunting back and forth. Isn't there a techinical term for this problem? I think you are supposed to set two differnt threshholds, one for going up and one for going down. Or use fuzzy logic or something.

Aside from that, I like the system. Oh, yeah, when you look at who rated a comment what, some of the handy links go away, like to the replies, so you have to back out to navigate properly. But that is sort of off topic.

Adequacy.org

I'll give you a 1, if you like... (4.00 / 10) (#14)
by Friendless on Wed Jan 10, 2001 at 05:36:51 PM EST

The new visibility of ratings hasn't changed the way I rate articles. I feel that K5 has a strong sense of doing what is right, e.g. vote the article up or down on its merits, vote the comment up or down in its merits. I don't think there is a lot of personal judgement, though all this gibberish about Anne Marie worries me - whoever the heck they are, let them be judged on the quality of their articles. I think as K5 members get to know each other better, there could be some personal problems, but hopefully they will be drowned out by the professional attitude of the masses.

The visibility of ratings has given me some ideas on how it might work better. Consider my comment on the story where farl thought he might have raped somebody. There were 12 ratings on that (so far), of which 5 were below 3, and 6 were above 3. Overall, the comment rated 2.91, which is mediocre, yet plainly there were people who thought I was right on the money and people who thought I was a dangerous sexual predator waiting to happen. I think the ratings reflected whether people agreed with me much more than they reflected the quality of the comment.

People's opinions are fine by me, but by expressing an opinion which met fierce opposition from some people, I lost my Trusted User status. I don't think it's correct that because I felt it was the right thing to do to support farl, that I should be penalised by the rating system. An alternative approach to calculating mojo might be to figure out the percentage of ratings which are above 3. If it is higher than 40%, you are trusted, otherwise you're not. That way, by writing well-argued opinions which are not necessarily popular, you can still succeed.

Making decisions by popular vote is democratic. Drowning out voices of dissent is not.

I don't see it quite like you do. (4.45 / 11) (#17)
by theR on Wed Jan 10, 2001 at 06:01:20 PM EST

First, let me say that everyone sees the ratings, including which direction they are trending, differently. Unless someone can provide accurate statistics, what good does it do for someone to say that accountability has caused higher ratings? This is anecdotal evidence, not a definite trend.

Also, even if it is affecting the ratings, does it really matter? I will restate my opinion that I have voiced many times, which is that ratings are mainly to take care of the spam and pick trusted and untrusted users. I generally do not use ratings to tell me what to read and I would hope that most people would do the same. The only comments where ratings mean much to me are the ones rated below one. For everything, though, I would rather read it for myself and make my own judgement as to the worth of the comment.

Let me address a few specifics now:

One, that the reward/punishment flavor that ratings seem to be acquiring will increase the feedback effect of the community, encouraging opinion-consensus and groupthink.

I don't see how allowing people to see who rated them has changed the reward/punishment aspect some people have towards ratings. There will always be people trying to get high ratings and do not want low ones. How has this changed with accountable ratings?

Two, that an increasing amount of time will be spent on think about and talking about the ratings themselves instead of letting them just be a mechanism by which posts are ordered.

People have been talking about ratings and the system since I started reading K5 (which was right after rusty took a month off and blamed it on DoS, heh). I don't think that will ever change, and the more people that are here, the more people that will talk about it. More importantly to me, ratings are not a mechanism by which posts should be ordered. This may seem fine to you, but not me. For now, kuro5hin is definitely readable without ordering the posts by ratings. There is a reason the default method of sorting comments is to ignore ratings.

Three, and perhaps it should have been one being the most clear and present concern, that ratings will become increasingly meaningless as more and more people use only 4 and 5 votes occassionally daring a 3.

It seems that all three of these points you have made say the same thing, which is that ratings are not achieving the goal of telling us how worthwhile comments are. I do not think that is what the ratings are for and you seem to be one of the ones falling into the punishment/reward trap. Ratings don't really matter. The only time they make any difference is when you are rated below a one, untrusted, trusted, or if you choose to order comments by rating. Ratings are just as meaningless now as they were before and I don't think accountability has changed that.



sometimes ratings matter (3.33 / 3) (#35)
by speek on Wed Jan 10, 2001 at 07:59:46 PM EST

On K5 where we mostly discuss fairly ambiguous and philosophical/belief issues, ratings don't matter much. One person's opinion on the subject is generally no more or less interesting than anothers. Certainly none are informative.

However, sometimes a real "news" story comes through, or a technicaly piece, and at those times, I appreciate sorting by rating because it helps me find the informative comments that tell me "this is bunk", or "this is vaporware", or "this is real, and here's why...". Sorting those knowledgeable comments to the top is extremely useful when it's a basically a dry technical story with 500 comments.

--
al queda is kicking themsleves for not knowing about the levees
[ Parent ]

K5 (3.66 / 3) (#39)
by theR on Thu Jan 11, 2001 at 12:11:26 AM EST

Of course I am speaking of K5 when I say ratings don't matter. It is hard to speak about ratings on sites in a general sense because there are so many different dynamics goin on. The example you use of a story with 500 comments does not yet apply here, though.

At the places it does apply, I think you might be giving the people doing the rating a little too much credit. Sure, some or even many might know whether the comment has merit, but I would point to Slashdot as a perfect example of a lot of people moderating comments about things technical that they don't really understand. I still believe that the best way to judge a comment is by doing it yourself instead of relying on ratings, even if it is not always the most convenient thing to do.



[ Parent ]
More to it (4.50 / 2) (#45)
by speek on Thu Jan 11, 2001 at 07:04:39 AM EST

I think you might be giving the people doing the rating a little too much credit

Let's talk about slashdot, because I find you are entirely correct about K5 where I've had no use for the ratings. On Slashdot, however, some news piece comes in about which I know nothing, yet I have some interest. Once 300-400 posts have come down and they've been rated, there's generally a number of highly rated posts. They don't all agree with each other, and there's often lively debate going on sparked by these posts. Reading those several posts plus their debates quite often is very helpful in coming to a better understanding of the news story. Without the ratings, I would have had to wade through a tremendous amount of crap, and my time and convenience are worth quite a lot to me.

Additionally, when I know little about a subject, I do put more trust in the raters. Collectively, 5 raters almost certainly know a lot more than I do, so their averaged score is usually more accurate than mine alone.

--
al queda is kicking themsleves for not knowing about the levees
[ Parent ]

Disagree mostly... (4.92 / 14) (#19)
by rusty on Wed Jan 10, 2001 at 06:06:04 PM EST

...and here's why:

  1. Weak foundation for the argument. The refrain I hear constantly when someone has a concern usually runs like: "I saw this one example of [problem foo], and I assume it must be a rash of epic proportions. What are you going to do about it?" This is not very convincing. Sometimes it really is a continuing problem, more often, it isn't. In this case, I also noticed the comment you pointed out, and I remember it because it was the only one of it's kind that I can recall seeing. That doesn't say "disaster" to me...
  2. Weak foundation for argument. This article also assumes that there are rating wars going on that are not noticable by the casual reader. Actually there's been exactly one such event, and a few simple emails convinced the parties involved that this whole thing was not worth their time or energy. Besides a couple of users who consistently rate badly and have virtually no effect on anyone, opening ratings hasn't led to any kind of dispute that we have been made aware of. If anyone is currently engaged in a rating war, please email help@kuro5hin.org and let us know.
Now, let's see if your assumptions are right. Yes, there has been a small uptick in average rating since ratings were opened. Lemme give you hard numbers. I ran this set of stats once on Nov 17, 2000 (before open ratings), and again today. Here's how it breaks down:

Average comment rating (of rated comments):

11/17/2000: 3.16
01/10/2001: 3.26

Comments per rating range:

11/17/2000:
0 <= rating < 1: 72 (0.29%)
1 <= rating < 2: 2613 (10.73%)
2 <= rating < 3: 5266 (21.62%)
3 <= rating < 4: 9038 (37.10%)
4 <= rating <= 5: 7370 (30.26%)

01/10/2001:
0 <= rating < 1: 128 (0.29%)
1 <= rating < 2: 3805 (8.71%)
2 <= rating < 3: 8801 (20.14%)
3 <= rating < 4: 16750 (38.32%)
4 <= rating <= 5: 14225 (32.54%)

So, good guess. Yes, there has been a slight upswing in high-end ratings. Notably, 2's and 3's are losing place, proportionally, to 4's and 5's.

But, first, this is a two data-point study right now, and thus doesn't tell us that much. I'll run it again periodically and see if things keep changing, and how.

Second, the interpretation of these numbers is very much up in the air. Does it mean that people are afraid to give low ratings, or does it just mean that people who, before, would give low ratings because it was secret have stopped doing that? I'm very much inclined to believe the latter, personally. I did expect that there'd be an upswing in the average rating, and that it wouldn't be enormous. That seems to have happened. But the breakdown doesn't, I think, support a hypothesis of massive rating inflation. basically 1-2 and 2-3 both lost 2 percentage points, which were redistributed in 3-4 and 4-5, while 0-1 has remained steady.

But interpret for yourself. Do people think this is a trend that should be watched? And more importantly, if you think ratings should not be public, what would be better? And how many of you actually refrain from rating low due to fear of reprisal, even if you think the majority of readers would agree with you?

____
Not the real rusty

Counterpoints & Comments (3.87 / 8) (#25)
by Parity on Wed Jan 10, 2001 at 06:40:59 PM EST

Firstly, not all subjective observation is 'anecdotal' ... the particular comment was anecdotal, (based on a single experience, after all), but the observation about the ratings trend was based on observing a statistically significant number of conversations. In other words, it was subjective and intuitive, not hard numbers, but neither was it precisely 'anecdotal'. Okay, that's a nitpick, I admit it.

Secondly, though, I don't assume that there are ratings-wars going on... I assume that the fear that a ratings war is possible can, will, and does influence people's actions. Whether or not such wars occur or not is immaterial, as long as people believe that they could be subject to reprisal.

Finally, yes, I'd like to see more numbers. I'd also like a better understanding of those numbers - are those on comments posted just on the date specified, that are held in the database on the date specified, or that are posted between the last time the analysis was run and the date specified... ? Statistics are only meaningful if you have the whole story.

Parity None

[ Parent ]
About the stats (4.20 / 5) (#30)
by rusty on Wed Jan 10, 2001 at 07:24:44 PM EST

Those statistics are comprehensive, up to that point in time. That is, the first set considers all rated comments up to 11/17/2000, and the second considers all rated comments up to 01/10/2001. So, the second set does include the first. As I said, I'll keep running this check every month or so0-- it ought to be interesting to see how things evolve. I may even automate it and put up a "stats" page so people can look for themselves.

I didn't mean to say that all anecdotal evidence is wrong, just that it's not very solid. I appreciate people keeping an eye on this stuff, definitely, since it can provide a good early warning of incipient problems. All I meant was that I don't think this is a problem yet, and I wanted to propose an alternative explanation of what the numbers could mean.

I think the possibility of "rating wars" does, and should, influence people's actions. I don't think anyone's gonna go nuts on you because you rated one of their comments "1". But if you do it to every one, no matter what the comment, well, that's gonna show up. The thing is, it works both ways. My advice is, if you think someone's rating unfairly, email help@kuro5hin.org, and we'll look at it.

Basically, though, ratings don't mean much, and it's an evolving system. The main gist of my reply was that you were partly right in your observations, but that I expected that to happen, and I don't think it's a problem. If it becomes one, I hope people won't keep quiet about it... but you (as a group) have shown no indication of keeping quiet about things so far. :-)

____
Not the real rusty
[ Parent ]

Put another way... or maybe the same? (3.00 / 3) (#41)
by theR on Thu Jan 11, 2001 at 12:23:53 AM EST

If I may, rusty, I believe the reason you changed the system in the first place was so people would be less likely to indiscriminately rate down people that they don't like, including the comments by that person that have merit. So, if this was successful, then of course average ratings would go up because fewer comments should end up rated a 1 or 0.



[ Parent ]
zero ratings (4.00 / 4) (#52)
by Refrag on Thu Jan 11, 2001 at 02:40:11 PM EST

The problem with how Rusty has implemented public ratings is that all ratings are public except for zero ratings. All he does is count the zero ratings for a particular comment. I feel that zero ratings are the ratings most likely to be abused.

Since public ratings have been enabled I have received one zero ratings (that I know of). The ratings occured on a fairly harmless comment that was intended as humorous. I can understand getting a rating of one on this comment, but I just do not understand why someone would go out of their way to rate such a comment with a zero. The comment got a few fives, obviously from people that found it humorous, and that zero really brings the average down.

Basically, I think that if Rusty wants the ratings to be public, he should fully disclose them.

Refrag

Kuro5hin: ...and culture, from the trenches
[ Parent ]

Wow (3.00 / 3) (#54)
by theR on Thu Jan 11, 2001 at 02:49:33 PM EST

I didn't even notice that you can't see who rated a comment '0' if you're not trusted. You're correct about that. Even if comments less than '1' can't be seen by everybody, everybody should be able to see who gave a rating of '0' to a comment that is not hidden.



[ Parent ]
Why 0's are hidden (3.50 / 4) (#56)
by rusty on Thu Jan 11, 2001 at 04:13:40 PM EST

I chose to hide the identity of zero-raters because those are the most likely to cause feelings of persecution, and retributive jihads. Basically, I don't want trusted users to be afraid to use the 0 rating because they'll just get "stripped" of their trust by the irate commenter.

There is very strong peer-review for zero ratings, mind you. If you're trusted as well, you do see who rated what 0. TU's also have a link to the recent hidden comments right there in their user box. Math says that one TU can overrule any four others about whether a comment should be hidden or not. So the balance of power is heavily stacked against abusive hiding. I felt that these factors were strong enough peer review to justify the extra stability of not revealing who exactly is trusted.

____
Not the real rusty
[ Parent ]

Yeah, but... (3.50 / 2) (#61)
by pb on Thu Jan 11, 2001 at 11:18:31 PM EST

First, I lost my trusted user status on a zero rating, and I'm pretty pissed that suddenly I can't see who it was. Also, I don't think the rating itself was terribly fair, considering the total score of the post didn't fall below one, and therefore will probalby never get evaluated for abuse.

Finally, what did you do to change the determination of Trusted User Status? I know you changed it, but it certainly doesn't work like it did before. Do you wait a fixed interval before re-evaluating users, or what?
---
"See what the drooling, ravening, flesh-eating hordes^W^W^W^WKuro5hin.org readers have to say."
-- pwhysall
[ Parent ]
upped the number of comments (3.50 / 2) (#62)
by rusty on Fri Jan 12, 2001 at 01:31:19 AM EST

The number of comments required used to be ten, now it's 18. There were an awful lot of trusted users, and it appeared, from looking at the list, that 18 was a pretty good cutoff line to tweak that number. Like, a lot of people had 10-12 comments, but maybe half as many were over 20. Judging from the last couple days (since the change happened), it seems to have worked pretty well.

____
Not the real rusty
[ Parent ]
Well, now I know... (4.00 / 1) (#63)
by pb on Fri Jan 12, 2001 at 01:37:06 AM EST

Ok, it wasn't just that, it was *two* different things that confused me, and now that I'm a trusted user again, I see what happened.

Essentially, three users rated the comments in a thread the same, (well, at the time it was probably the whole thread) but one gave it a 3, one gave it a 1, and one gave it a zero; that's where I got my mystery zero.

However, I was doubly confused after I lost trusted user status because I couldn't find my post in the story at all, even though it was rated above 1.00. That's because the parent comment had a rating below 1.00, and it took the entire thread with it! That's probably also why no one modded my comment back up: the trusted users didn't see it on the list of Hidden Comments (although it was), and the untrusted users didn't see it at all!

I can see why you don't reparent the comments, because that would be confusing. But some comments practically end up in limbo. Does that qualify as a bug? :)
---
"See what the drooling, ravening, flesh-eating hordes^W^W^W^WKuro5hin.org readers have to say."
-- pwhysall
[ Parent ]
limbo comments (4.00 / 1) (#68)
by rusty on Fri Jan 12, 2001 at 12:20:37 PM EST

When a comment is rated 0, the whole thread disappears for two reasons. The major one is that if a comment is spam, then responses to it are probably useless anyway. It's part of the "encouraging people not to respond to crap" strategy. The second is that if just the original comment went away, you'd have responses to it floating up and probably making no sense whatsoever. So, that was done on purpose.

____
Not the real rusty
[ Parent ]
I don't like that, but it's interesting... (3.00 / 1) (#71)
by pb on Fri Jan 12, 2001 at 01:39:11 PM EST

Then could you have them show up as Hidden Comments?

(or at least have them show up *somewhere*? Or make it an option...)
---
"See what the drooling, ravening, flesh-eating hordes^W^W^W^WKuro5hin.org readers have to say."
-- pwhysall
[ Parent ]
Conceptual bug? (3.50 / 2) (#75)
by Estanislao Martínez on Sat Jan 13, 2001 at 10:20:16 PM EST

Math says that one TU can overrule any four others about whether a comment should be hidden or not.

This reasoning I think is flawed in a fundamental manner.

IF somebody gave a comment a 0 unfairly, sure, I can give it a 5, and it will withstand 3 more 0 mods without going under one. But if I were to rate that way, I would be rating on the basis of getting the comment to a certain score, not on what I think the comment should deserve to be rated.

Here is the limiting case. Suppose there is a comment which pretty much everybody thinks deserves a 1. It is inane, but not spam, grossly offtopic or offensive crap. Somebody gives it a 0. If I were to rate this comment what I believe it deserves (1), I can't bring it back up. Even more, if apart from that one TU abuser, the other ratings to the comment average 1 (the deserved score), the rating will never rise to 1.

So in at least one case 0-ratings break the idea that you should rate a comment what you believe it deserves.

--em
[ Parent ]

Ratings & Ratings... (none / 0) (#77)
by Parity on Mon Jan 15, 2001 at 03:55:57 PM EST

I think that while it's good to give a comment the score it deserves as much as possible, that it's not unreasonable to break from that policy in the case of countering an abusive use of the 0 comment, especially not when it's something like giving it a 2 or a 3 to make a comment visible instead of giving it the 1 or 2 that you thought it deserved but you didn't want to leave it at 0.50 or precisely 1.00.

Anyway, there have certainly been times where I voted 3 on one comment and 4 on another comment when I thought the comments were about equal... at 3.5ish... but of course, you can't vote '3.5', but you can vote '3' on a post that is currently rated 4.16.

Parity None



[ Parent ]
OK, but... (3.00 / 5) (#26)
by itsbruce on Wed Jan 10, 2001 at 06:41:18 PM EST

I don't think 11223's comments were intended to mean anything inquisitorial but I do think that the implications I saw in them are valid.

Before the exposed ratings, I trusted the overall results and didn't worry about them. Now, I'm going to wonder.


--

It is impolite to tell a man who is carrying you on his shoulders that his head smells.
[ Parent ]
I tend to agree (3.28 / 7) (#28)
by SEAL on Wed Jan 10, 2001 at 06:58:50 PM EST

Does it mean that people are afraid to give low ratings, or does it just mean that people who, before, would give low ratings because it was secret have stopped doing that? I'm very much inclined to believe the latter, personally.

The discussions over highly controversial topics really make this shine through. Consider, for example, debates on gun control or U.S. vs. the rest of the world type stuff, esp. the military. You can make the world's most thoughtful, well constructed post in such a discussion. Invariably you will get a large number of people rating your post to 1 because they don't agree with the stance you are taking.

I think the open ratings have alleviated a little of that. People that abuse ratings in this manner become very obvious. A couple clicks and you can usually see a string of their comments which disagree with what you said, or even a more general trend of philosophical differences across multiple topics.

Obviously that doesn't stop them from rating you, but it's easy to point out. The writeup for this topic viewed "meta" comments as distasteful, but I don't think it's bad to point someone out if they are rating you based on personal disagreement.

On a good note, it seems most people here are pretty restrained in thier use of the 0 rating.

SEAL

It's only after we've lost everything that we're free to do anything.
[ Parent ]

Eloquence's rating sums it up... (4.50 / 6) (#38)
by SEAL on Wed Jan 10, 2001 at 11:53:28 PM EST

The topic IS meta, so I'll continue here :)

Eloquence's rating on my parent post was done because he disagrees with the content of that post. Read this post for details.

An excerpt:

This leads me to rate comments as follows: 5 if I agree completely, 4 if I agree except for some minor point, 3 if I agree with some parts or don't care much, 2 if I disagree with most of the content, 1 if I disagree with all of it. I think I have never used 0 to vote something down just because I disagree.

First things first: Eloquence, I think my post deserved a 2 under your system, unless you truly believe the 0 rating is being abused :)

More seriously, is the reason I don't like seeing posts rated low simply because the rater and poster disagree. I think it tends to cover up dissenting views in heated discussions. Of course, you can set your preferences to Oldest first. But the point is that a higher rated post tends to be seen by more people. Eloquence's post acknowledges this when he states:

The reason for this [rating dissenting posts down] is that I want the rating system to allow users to find valuable information quicker.

K5 is a forum for discussion. Information dissemination, in my opinion, is secondary. If someone posts and fabricates information (essentially lying), then I would rate that post down, dissenting opinion or not, with a few exceptions for humorous posts and such.

Rating down informative posts that present differing opinions detracts from discussion, though. A good discussion should include various viewpoints and allow people to hear all sides of a story. This helps us become more knowledgeable, and may even change peoples' minds. A debate in my mind is a good thing, as long as it is not just a mindless flamewar.

Fortunately this is a minor issue, since the rating system doesn't hide posts, except at the sub-1 level. That doesn't change my opinion of how posts should be rated, though.

Consider: with the post I linked above, I would've rated it high, even though I disagree with it, because it makes people think. It didn't change my mind but it could have. Eloquence, in my shoes, would have rated it down and been on his way.

Best regards,

SEAL

It's only after we've lost everything that we're free to do anything.
[ Parent ]

Significant difference? (3.00 / 2) (#70)
by tmoertel on Fri Jan 12, 2001 at 12:46:23 PM EST

11/17/2000: 3.16
01/10/2001: 3.26
Can you compute the standard deviations as well? I'm wondering if those differences are even statistically significant.

--
My blog | LectroTest

[ Disagree? Reply. ]


[ Parent ]
Well... (3.21 / 14) (#20)
by FlinkDelDinky on Wed Jan 10, 2001 at 06:21:21 PM EST

I've triad to make a run at 'trusted' status but just failed because, I think, I like plasma nuclear reactors for power generation.

I did rate less, I tended to rate what I thought was fair but didn't rate if I thought it was below a three for fear of retaliation.

Now that I've come to realize that K5's rating scheme is stupid and most trusted users are fucking idiots with no sense of humor I'll be participating as my normal, intelligent, extremely charismatic self again.

Unfortunately many of you may not get to enjoy the posting excitement that is flinkdeldinky because the trusted users have there heads up ther ass.

What has made K5 so god damn elitest. Posting on any devisive topic can ruin ones trusted status. Rating properly can ruin ones trusted status. It's all bullshit!

Since I'm sure that I'll never get 'trusted' and since I've concluded that the rating system has been sabatoged because of its link to 'trust', why shouldn't I engage in retaliation rating?

PS. Hey I've got an idea, anybody that rates this as a +5 I'll go and rate there most recent prior post a +5. If enough of us join together we can become a 'trusted' pack! Then we can 0 rate anybody who doesn't rate us 5.

Funny, but a real danger (3.14 / 7) (#22)
by pope nihil on Wed Jan 10, 2001 at 06:25:36 PM EST

Score 5

:)


I voted.

[ Parent ]
You liar;) (2.33 / 3) (#27)
by itsbruce on Wed Jan 10, 2001 at 06:49:23 PM EST

Score 5

At the time I write this, you've rated him 4! Or is this a trap? All you have to do is change your rating to 5 and make me look silly;)


--

It is impolite to tell a man who is carrying you on his shoulders that his head smells.
[ Parent ]
Well, (2.00 / 3) (#36)
by pope nihil on Wed Jan 10, 2001 at 08:01:49 PM EST

Originally, I rated your comment a 5. Then I changed it to a 4. Now I've changed it back. Unfortunately, there is no feature for editing comments.


I voted.

[ Parent ]
Actually... (4.20 / 5) (#23)
by Parity on Wed Jan 10, 2001 at 06:29:06 PM EST

If this comment is typical of your posted offerings, it may be the high-handed 'I'm right and everyone else is wrong' tone and/or the gratuitous profanity, both of which conspire to make even a relevant, on-topic comment look like a troll to the casual observer.

Not to mention that voting circles are in direct controvention of the purpose of ratings (not to mention another possible problem with exposed ratings that I didn't think to include in my article.)

That said, yes, K5 tends to be a bit stuffy, a bit without humor, and a bit concerned about proper language. All this is probably in reaction or over-reaction to the evolution or devolution of slashdot.

Parity Even

[ Parent ]
Wacky (4.16 / 6) (#33)
by rusty on Wed Jan 10, 2001 at 07:40:50 PM EST

Wow, you need to really not take this so seriously.

Ok, apparently the majority of users here don't particularly value humor in comments. Is that such a bad thing? When you read a story that isn't humorous in intent, do you want joke replies to be the first thing you see, or maybe among the last? I personally think that humor is good and all, but probably shouldn't be the most attention-getting comments.

Now, about your ratings themselves, let's see who actually rated the comments you point to. The first one has no zero ratings, and the second has two out of 24 (those zeros, by the way, are very very much undeserved. Please, people, that comment is *not* a zero! Read the guidelines. Sheesh). So you just went on a tirade against "trusted users" based on the actions of two (out of more than 200) of them.

What has made K5 so god damn elitest. Posting on any devisive topic can ruin ones trusted status. Rating properly can ruin ones trusted status.

This is incoherent ranting. We have an article complaining that ratings are being *inflated*, and even numerical evidence of this, but here you are complaining that posting anything will lower your ratings. And obsessing over trusted status is just silly. You want it that bad? Make a diary, post 18 comments, make a new account, rate them all 5. There. Enjoy. It's just not something that's worth getting this excited about.

Since I'm sure that I'll never get 'trusted' and since I've concluded that the rating system has been sabatoged because of its link to 'trust', why shouldn't I engage in retaliation rating?

Or, to paraphrase, "a bunch of people didn't think my slightly funny one-liners were pearls of wisdom from heaven, so I'm perfectly right to go off and do whatever I please." C'mon, man, I know you're more mature than that. What would you be accomplishing? What would the point be? Jesus, you're user 172-- you've been here a long, long time. You know better than this.

____
Not the real rusty
[ Parent ]

Okay (3.00 / 1) (#46)
by FlinkDelDinky on Thu Jan 11, 2001 at 08:51:07 AM EST

Wow, you need to really not take this so seriously.

No, I was just making a troll style rant because K5's becomming a hangout for prigs.

When you read a story that isn't humorous in intent, do you want joke replies to be the first thing you see, or maybe among the last?

Actually, I could care less. I wanted something fun and ranty to write. But now that I think about it, I'm getting lots of attention with an inflammatory rant. Besides, some of these articles are jokes, so why not pile on?

So you just went on a tirade against "trusted users" based on the actions of two (out of more than 200) of them.

...No, I went on a tirade because of this...

Make a diary, post 18 comments, make a new account, rate them all 5. There. Enjoy. It's just not something that's worth getting this excited about.

Trusted is stupid. I'm not particularly excited about this, I just don't like that trusted is so completely stupid. I think we should all be equal except for the administrators.

Trusted annoys my sense of commraderrie (<-- I could look that up and appear intelligent but I'm to lazy for that).

[ Parent ]

Anger management (3.00 / 1) (#55)
by rusty on Thu Jan 11, 2001 at 04:08:20 PM EST

No, I was just making a troll style rant because K5's becomming a hangout for prigs.

Y'know, right after I finished writing the first response, I thought "hmmm. IHBT, maybe."

Trusted is stupid.

"stupid" is not a meaningful criticism. I can't fix a problem because it's "stupid". There's no code in there that says "while ($trusted_user eq 'stupid')" that I can just go clean up. Basically, if you want to whinge, whinge meaningfully.

It would be nice for us all to be equal. But do we do that at the price of ignoring spammers, and just letting them spam? 2 admins cannot deal with all possible abusive comments. That's the point of "trusted", soleley and completely. It's a first line of defence against comment spamming.

I think basically you're reading way, way more into it than is actually there. "Trusted" here is a euphemism for "has to see the crap posts, will feel obligated to do extra-attentive rating for the rest of us". It's a service position. It's not a privilege, and it doesn't make anyone better than you. And I dispence advice on how to become trusted easily to convince people that that is true. Basically, I want you to try it and see. Every time someone has done so, and they discover what "trusted" really is, they forget all about it and go back to being their normal selves.

The reason it works is because people who become trusted without trying tend to be committed and frequent enough visitors to actually do the extra work that is requested of them.

If you're ok with the idea that admins don't have to be "the same as the rest of us", why can't you accept that trusted users are just admins with very very little extra privilege? Would it help if I called them "Santa's little helpers" instead of "Trusted users"?

I suppose I should give up and let you believe whatever you want. But IMO, this is important. You've got it all wrong here, and you're believing that things work in a way I specifically designed them *not* to work. I wanted to keep "trust" from giving anyone more power to be heard, which is why trusted users post with no default rating, just like the rest of us. Their rating carries no more weight than anyone else's, and one trsuted user can reverse the actions of any four others (do the math-- if four TU's rate a comment "0", and one comes along and disagrees and rates it "5", that comment is now a "1", and remains visible to the site as a whole). So it's extremely stricly reviewed, and very weakly privileged.

Why are you so angry?

____
Not the real rusty
[ Parent ]

I hate it with these titles (3.00 / 1) (#60)
by FlinkDelDinky on Thu Jan 11, 2001 at 08:50:11 PM EST

"stupid" is not a meaningful criticism

You know, you're right. But my thinking isn't that the code is broken it's that the concept is broken

It would be nice for us all to be equal

Sure but now you've got a bunch of Signal 11 mojo lovers acting as 'Service Administrators'. You don't seem to realize that you can make a worthwhile (3 or better) post that goes against the grain of the K5 community and gets modded below a 3. In fact if you're a good poster you'll do that often, stirs up the pot.

elenchos's complaint points this out clearly. One day he's 'trusted' to moderate 0. Suddenly, he makes a good but unpopular post so he's no longer capable of judging a particularly nasty post?

Would it help if I called them "Santa's little helpers" instead of "Trusted users"

Would you mind? How about 'Service Administrator'? And while your at it why don't you throw out the whole 'trusted' concept and simply designate K5's hardcore as 'Service Administrators'. That way you can avoid these petty (but kind of fun) flameuments.

I actually don't want to moderate 0 as I've never seen one here. But you stuck a carrot in front of a bunch obbssesive-compulsive geeks didn't you?

And I'm not angry GOD DAMMIT :-) (<-- I really should use those more often, there absence has caused trouble all the way back in the BBS days of RIME net, UN'I, and the rest. But then I figure it takes some of the humor away)

IHBT, I Have Been Taken?

BTW, I really do enjoy your site. Once you get story edit I might actually become obssesive-compulsive enough to submit again.

[ Parent ]

Smiley Management and cabalism (3.00 / 1) (#69)
by rusty on Fri Jan 12, 2001 at 12:29:06 PM EST

One day he's 'trusted' to moderate 0. Suddenly, he makes a good but unpopular post so he's no longer capable of judging a particularly nasty post?

This is an unfortunate byproduct of the system being reactive to current state. Some people are hovering right on the edge of "trusted", and one or two unpopular comments can change that. I can't think of any good way to ensure that the system is reactive, but that trusted users stay trusted no matter what. I think, in fact, that it would be worse to make it stable at the expense of being responsive (a la Slashdot).

And while your at it why don't you throw out the whole 'trusted' concept and simply designate K5's hardcore as 'Service Administrators'. That way you can avoid these petty (but kind of fun) flameuments.

Three reasons I don't just designate by executive order.

  1. I have better things to do with my time. :-)
  2. I don't want the responsibility of choosing who is "good", and watching over them all the time to be sure they're still "good".
  3. If "Service Admins" were hand-picked, we would just get accused of bias and cabal-ism. I'd rather have the system take the blame than me personally. At least this way, You (collectively) do control who has service admin privileges.
Does that make sense?

And I'm not angry GOD DAMMIT :-)

Ok. :-) You have to admit, your first post sounded really angry. It read that way to me anyway. You do need to use more smileys.

And IHBT == I Have Been Trolled.

____
Not the real rusty
[ Parent ]

Okay (1.50 / 2) (#74)
by FlinkDelDinky on Fri Jan 12, 2001 at 06:51:06 PM EST

Does that make sense?

Yes.

PS. Bring back the photo of the bridge on at the top of the page. Driph's done great things but that icon isn't one of them.

[ Parent ]

name change (3.00 / 1) (#73)
by superfly on Fri Jan 12, 2001 at 05:14:43 PM EST

It would be nice for us all to be equal. But do we do that at the price of ignoring spammers, and just letting them spam? 2 admins cannot deal with all possible abusive comments. That's the point of "trusted", soleley and completely. It's a first line of defence against comment spamming.

How about calling them "spam killers" if that's all they're for? That might stop some people from getting too worked up about it.



[ Parent ]
+5 for the last bit... (3.00 / 2) (#42)
by eventi on Thu Jan 11, 2001 at 12:35:46 AM EST

I've been thinking along those lines, ever since rusty deleted an accidental duplicate posting, and requested that the 2 zeros be changed from a rating I got. rusty gives me a 4 to try and even things out, and some guy, who obviously understands Jersey humor, gives me a 5. I immediately turned around and gave his most recent a 5, just to say thanks. Yeah right, no cabal...

[ Parent ]
Why I Like Accountability in Ratings... (3.50 / 8) (#24)
by eskimo on Wed Jan 10, 2001 at 06:40:38 PM EST

First of all, I like seeing how many people have rated my story. I am not as math intuitive as a lot of you guys (I can do the basic stuff...tips, batting averages over a short period of time, etc.), so it is helpful to not have go into algebra mode to see how many rated it.

Second of all, I like to see how people look at my stuff. I'll take half fives and ones over all threes anytime. Do I understand why they vote? Nope. Do I wish they would reply? Yep. But they don't. Oh well. I'll take one reply over a hundred ratings. Actually, that would be my only suggestion, and not an emphatic one either: make a response automatically worth some sort of rating.

In short though, even though I see names, they are no more accountable than they were before. If people are graphing or tracking this stuff, then maybe they have a little too much time on their hands. And I have a lot of time on my hands, so I still don't understand.

I am my own home. - Banana Yoshimoto

Unforeseen Joke (4.00 / 1) (#29)
by eskimo on Wed Jan 10, 2001 at 07:20:56 PM EST

Now you are all going to rate this '3.' Funny shit.

I am my own home. - Banana Yoshimoto
[ Parent ]

Two sides (4.28 / 7) (#31)
by leviathan on Wed Jan 10, 2001 at 07:25:58 PM EST

There are two sides to every opinion that you can voice, and this is one of those awkward ones where I can not only see both sides, but I agree with them both.

Against exposed ratings are the facts that, like you said, it's possible for discussion about ratings of discussion to break out, also the fact that I almost did something I shouldn't have a few minutes ago.

I saw yankeehack's story which I was about to moderate. I remembered some point where yankeehack rated me, in my opinion, unfairly lowly. It was me pointing out a typo in an editorial comment (and I'm not going to give you a link, you lazy sods), and I got a 1 rating for my troubles. That's yankeehack's perogative, and I don't think anyone ever said k5 ratings had to be fair.

More importantly, I don't really give a fsck about mojo. I'm not trusted, and by the looks of it I ain't never gonna be neither. There is only one way that I will promote groupthink, and that way it to try and educate everyone round to my way of thinking by means of a reasoned argument. I'm certainly not going to argue for something I don't agree in just to get mojo.

So, I saw yankeehack's story. I was going to vote it -1; not because I thought it was so badly written that it should be resumbitted (although I do think it could be written a better), and not because I thought it was a pointless story, but because it was written by yankeehack. You'll note that I didn't in fact, but the thought did seriously cross my mind.

For exposed ratings is a matter of having seen the good it has brought. People can no longer apply their 1 or their 5 to their worst enemies or their best mates (or themselves; how narcissistic is that?) with impunity. There's not a great deal we can do about it directly although knowledge is power (she says, trying to prop up a weak argument). There's also the rather more damning evidence that rusty has come up with, of course.

So you see, on one hand I know that I have the self-control to not resort to carrying out personal vendettas, and on the other hand that's the only thing that we can do if we find we're being unfairly rated. Complaining in a meta-comment isn't going to get you very far.

--
I wish everyone was peaceful. Then I could take over the planet with a butter knife.
- Dogbert

but can't it be corrected by others??? (3.00 / 1) (#59)
by yankeehack on Thu Jan 11, 2001 at 05:46:44 PM EST

Hopefully this will be answered, but here goes...

If my mod (or anyone else's for that matter) is unfair, wouldn't it stand to reason that someone else will come along and "correct" it to the true postion, either up or down? Which in the case you cited, arguably did happen?

Personally, I don't mod most comments, just those that make an impression on me.

I can understand worrying about a mod in which a only few selected bunch *can* mod, but everyone on K5 at least had the chance to see the mod and vote on it themselves....

Again, it wasn't against you personally.... ok? :-)

No one who was bad in bed has ever been good in life (i.e. liberals, I've never had sex with a liberal woman who knew how to use her body.) Keeteel :-P I'm *right*!
[ Parent ]

No worries (3.00 / 1) (#66)
by leviathan on Fri Jan 12, 2001 at 11:16:33 AM EST

My point was not that sure, in time a vaguely active comment will get an average of ratings, and excessively low and high ones will be cancelled out by people too, but was that in the short term it bothered me enough to consider voting differently on your story. That could only happen because moderating is now public.

And like I said, it's your perogative to mod how you choose and when I sit down to think for a second, I really don't care if one of my comments gets a low rating. It really doesn't bother me. What I am concerned about is that it could bother some people who would then vote differently on stories (and so inflict their retaliation on everyone).

--
I wish everyone was peaceful. Then I could take over the planet with a butter knife.
- Dogbert
[ Parent ]

Yes (4.00 / 2) (#67)
by ucblockhead on Fri Jan 12, 2001 at 12:18:02 PM EST

I've seen exactly this happen. In my own case, I voted (dump it) an article by a very famous member here for what I felt were very good (and objective reasons) only to have him immediately return fire by giving my explanatory post a 1. I could have gotten mad, but there's really no point in that my comment quickly got other, higher votes and the article got voted down anyway. The system, in my mind, worked.

There's really no point in getting mad about single votes. A negative vote can come from so many reasons. The moderator may misunderstand what you mean, might be having a bad day, have hit the wrong key, may be sauced out of his gourd, have had his account hacked or may just plain disagree with you. In the end, it'll all work out. I only care if I get multiple low votes on a post, and then I don't get mad, I try to figure out where I screwed up.

In the end, I like being able to see the ratings because, as so many others have said, it is another information channel. If someone whose opinion here I respect greatly deals me a 1, I'm damn well going to think about the post a lot more than if some luser who has never commented does the same. In a lot of ways, that is more important to me then the average rating because the opinions of people who I think are smart and insightful are a lot more meaningful than the opinions of some anonymous, random people to me.


-----------------------
This is k5. We're all tools - duxup
[ Parent ]

Standard Ratings (3.83 / 6) (#32)
by mattyb77 on Wed Jan 10, 2001 at 07:29:10 PM EST

I apologize if someone else has already made this suggestion.

Firstly, I see the need for a ratings system. K5 may very well get very large some day and ratings is the best way to filter the muck and garbage. I like the fact that it is the users that rate and not moderators.

With that said, I believe that the meaning of each number from 1 and 5 can vary from person to person. This makes me think that if we define each number then we be able to curve ratings inflation and the system will become more accurate.

Taking the author's example, here's something that might work for everyone.

5 = Excellent
4 = Very Good
3 = Fair
2 = Not So Good
1 = Inappropriate

Something like that might work.

Thoughts? Criticisms? Donuts?

--
"I bestow upon myself the `Doctorate of Cubicism', for educators are ignorant of Nature's Harmonic Time Cube Principle and cannot bestow the prestigious honor of wisdom upon the wisest human ever." -- Gene Ray, the wisest human ever
One step further (3.50 / 4) (#51)
by lmnop on Thu Jan 11, 2001 at 12:24:54 PM EST

After you define the numbers take them away. Just use the definitions of Excellent, Very Good, and so forth.

This will allow the site maintainers to play with the numbers w/o changing the functionality of the rating system.

-lmnop
"If all else fails, immortality can always be assured by spectacular error." -- John Kenneth Galbraith
[ Parent ]
The 'official' rating guide... (3.33 / 3) (#53)
by Parity on Thu Jan 11, 2001 at 02:45:13 PM EST

Inoshiro point out in his comment that there is already a ratings guide for us to refer to. Obviously I didn't know about that before, even though I had thought I had read the whole FAQ.

Parity None

[ Parent ]
The Contradiction (3.60 / 5) (#34)
by kagaku_ninja on Wed Jan 10, 2001 at 07:43:11 PM EST

I like the fact that the count of ratings, as well as the list of rating values are exposed.

Exposing the identity of raters is only meaningful if K5 wants to encourage retaliation. I suppose a less radical response to unfair ratings is to petition the admins, but then they can probably analyze the data and detect abuses if they were so inclined.

My thoughts, abridged (3.77 / 9) (#37)
by Eloquence on Wed Jan 10, 2001 at 09:09:45 PM EST

(Just wrote a long comment when machine froze, still haven't found cause for these constant crashes in spite of months of searching&trying. Agh. I'll try to reconstruct my most important thoughts keyword-style.)

  • As pointed out here, I refuse to be labeled a "K5 coward" or "rating abuser" just because I rate based on whether I agree or not. Ratings should reflect the truthfulness of a comment, not its eloquence.
  • Current system might actually avoid flame-wars because posters see who rated them down and not just the little big number. Remember a lot of postings at beginning of new rating system of the sort: "Oh, I'm relieved, now I know who rated me down."
  • Indeed the current system may reduce honesty of ratings, but in general, users will be inclined to be careful with low ratings because of the way the may emotionally affect authors. See Renderotica: lots of good ratings but nearly no bad ones -- fear of discouraging artists.
  • Current system may be close to maximum usability. See little room for improvements, perhaps optimize sorting to take into account number of ratings.
  • Best rating system may be one of +1,-1,+5(1 per story or day),-5(1 per story or day, more for trusted users). Calcualte overall score and sort by it, every user can set their individual hide threshold ŕ la Slashdot. (This would effectively make -1 the rating for comments that users want to be hidden, i.e. trolls.) Having less room in individual ratings may be a good thing that avoids useless meta-debate.

--
Copyright law is bad: infoAnarchy · Pleasure is good: Origins of Violence
spread the word!
Reasons for rating? (4.50 / 2) (#40)
by flieghund on Thu Jan 11, 2001 at 12:21:49 AM EST

First, we have this: "...I rate based on whether I agree or not." Then, we have this: "Ratings should reflect the truthfulness of a comment, not its eloquence." I think you've hit upon the three of the four ways I can think of to rate comments:

  1. Personal agreement (Is this what I believe?)
  2. Accuracy (Is the information presented correct?)
  3. Eloquence (Is the comment well-written and/or easy to understand?)
  4. Relevance (Is the comment meaningful to the discussion at hand?)

I'm not trying to say that these are the only possible motivations; these are merely the ones that I can think of, and that motivate me when it comes time for me to rate a comment. That being said, I try to avoid injecting #1 into the equation; unfortunately, that tends to take the shape of not voting on comments I disagree with.

Given your mention of numbers 1-3, I have some questions. The first part of your comment I quoted above strongly implies that you rate comments based upon your personal agreement. However, the second quoted part states your belief that comments should be rated based upon accuracy, rather than eloquence. How do you reconcile this discrepancy? Or do you believe that you are the ultimate arbiter of Truth?

Along these lines, how do you rate comments that do not contain facts, but rather only conjecture or opinion? Unless I'm lying, "I believe X" is always going to be a truthful statement because it is, in fact, what I believe. Yet if you do not agree with my stated opinion, this creates a conflict: do you rate low, because you do not agree, or do you rate high, because the comment is truthful?

I'm not trying to pick a fight or anything. I'm just curious if you really only rate based upon personal agreement, or if you apply a broader range of criteria. I believe that limiting ratings to a single basis has the potential to severely limit the quality of discussion.

(On a bit of a tangent, I think that's why there aren't more humorous posts made to kuro5hin. Everyone is so damned serious because they're afraid they'll get rated low if they don't make a deep, insightful comment that is well written and absolutely on-topic. Bah. Humor is the spice of life. Laugh a little.)


Using a Macintosh is like picking your nose: everyone likes to do it, but no one will admit to it.
[ Parent ]
No contradiction (2.50 / 2) (#43)
by Eloquence on Thu Jan 11, 2001 at 12:57:02 AM EST

Of course I believe that my opinions are correct and accurate. That's why I hold them. Of course I know that I am not "the ultimate arbiter of truth". So if a comment raises a point that is important and that I haven't thought about I either change my opinion and rate the comment up strongly, or I try to examine the point first in more detail and ignore the comment for the time being, or, when the issue is not that important, I ignore both the point and the comment.

However, if a comment makes a point which, from my point of view, is simply wrong -- logically or factually -- I will vote the comment down and/or reply to it, depending on its perceived importance.

The behavior that is very much criticized is perhaps something different: Rating something down because you just feel it's wrong but can't articulate why. But this is not really agreement/disagreement to me. I try not to say "I disagree" when I don't really know if I do. However, assuming that someone who votes your article down doesn't have any arguments against it (and therefore accusing him of "rating abuse") is presumptuous to say the least.

So, for me, personal agreement and accuracy are really the same. If something is inaccurate, I disagree, if it is accurate, I agree. If I don't know, I don't know.

Now, I can understand that people may want a comment instead of a rating, however in the interest of better sortability of comments, I rate comments even though I don't have the time to reply.

Statements of the sort "I believe X" are not really much different from statements of the sort "X is true", only in terms of confidence and/or courtesy of the speaker. For weaker statements, I expect weaker evidence, but otherwise I apply the same standards. Do I think that the poster is right in his belief? Do I have arguments against it? Is it logically convincing, would I agree with it even if I wouldn't already hold the belief in question?

Last but not least, there are entirely subjective comments like in the "Who are you" threads. I usually find rating them pointless, eloquence may indeed be of relevance here.
--
Copyright law is bad: infoAnarchy · Pleasure is good: Origins of Violence
spread the word!
[ Parent ]

Humor (3.50 / 2) (#44)
by Eloquence on Thu Jan 11, 2001 at 01:03:32 AM EST

Forgot that, but it's a different topic anyhow. I agree that we have a little humor deficiency on K5. The problem of moderators not getting the joke is greatly amplified when everyone's a mod. OTOH, I'm happy that we're not as crazy as Slashdot, where everyone tries to get rated +5:Funny. Especially with postings of little discussion value, or where the key questions are too difficult for most readers, people try to be funny at all cost. Need I say beowulf cluster?

I guess the best way to avoid being rated down is making clear your post is meant humorously. You don't need to use the <humor> tag, but a smiley here and there may help.

Oh, and the K5 crowd may actually have less humor.
--
Copyright law is bad: infoAnarchy · Pleasure is good: Origins of Violence
spread the word!
[ Parent ]

Following your own advice... (3.00 / 1) (#76)
by kubalaa on Sun Jan 14, 2001 at 02:37:47 PM EST

...I'd have rated this comment a 1, since I disagree with every one of your points.
"I rate based on whether I agree or not."
Truth is relative. If it weren't, k5 wouldn't be a *cough* discussion group, it'd be a news site. The purpose of rating is (unarguably) to eliminate trolls and direct those comments which are of interest to the majority of the readership to the top of the fray. Whether a comment is interesting to someone else has no relevance on whether you personally think it is true or not. Finally, if you disagree with a comment, you'd serve the community better by explaining why than by trying to "hide" the offending comment with a low rating.
"Current system might actually avoid flame-wars."
Were flamewars a problem before? Please enlighten me on how you can start a flamewar with an unknown individual. It's not as if people were ranting to nobody in particular about getting a low score.
"Users will be inclined to be careful with low ratings because of the way they may emotionally affect authors."
This is a GOOD thing? Give me a break. There's a place and time for sensitivity, and this is not it. Anybody who is that dependent on their mojo for a sense of self-worth has more problems than me rating them down. And the fact is that this is not a support group where people come to feel good about themselves, it's a discussion group where people come to read interesting comments. I'm sorry if it hurts the author's feelings, but if a comment is not interesting, then it doesn't deserve a good rating.
"Current system may be close to maximum usability."
It's usable. Maximum usability means I never have to read a single comment that doesn't make me think. We're a long way from that.
"ŕ la Slashdot"
Did you just hold up Slashdot's moderation system as worthy of emulation? I'll restrain my laughter.


[ Parent ]
Raise your hand if... (3.50 / 4) (#47)
by jabber on Thu Jan 11, 2001 at 10:09:56 AM EST

Who actually cares about the ratings their posts receive? If you post something you consider to be worthwhile, great! If others do not agree, fine. If you know who they are, and see from their own posts that they are raving loons, all the better. If someone who's opinions and knowledge you respect rates you down, learn from it.

Mojo is a very poor means to determine self-esteem. You're better off with a couple of dice. Having comment ratings be un-anonymous is a Good Thing. It keeps people like tewl and Latrene Spratwell (or whatever his nick is) from getting a cheap high off of vindictive jihads while providing a solid sounding board to ideas when rated by people who have earned a good reputation.

It's like the recent comment by iGrrl, tearing into the "Anne Marie" story on dementia and popular music. An authoritative voice can make or break a story, and the same should apply to comments as well.

[TINK5C] |"Is K5 my kapusta intellectual teddy bear?"| "Yes"

Defending myself (3.00 / 1) (#64)
by tewl on Fri Jan 12, 2001 at 10:45:09 AM EST

I'm sick of doing this. "People" like me? I'm constantly being reminded by my boyfriend to not leave my account logged in at work so something like that doesn't happen again. Jeez, take a look at my comments, and you'll find the truth.



[ Parent ]
Change your nick (3.00 / 1) (#65)
by StrontiumDog on Fri Jan 12, 2001 at 10:55:10 AM EST

it's obviously causing you some grief, and there is no harm in changing it to something else, like "kewl", "StrontiumBitch" or "uGrrl". The main reasons people maintain nicks is (1) they like the sound greatly or (2) it has built up a reputation they don't want to lose. You obviously don't like the reputation, so (2) can't be a problem. I can see you liking the sound of "tewl", but I'm sure you can think of an equally good nick. Need any help, just ask -- I'm brimming with suggestions.

[ Parent ]
agreed (3.00 / 1) (#72)
by mikpos on Fri Jan 12, 2001 at 04:56:06 PM EST

The ratings on k5 have about as much meaning as those on Am I Hot or Not?: basically a way to express your current emotional state via a nice click'n'drool interface. Since they have no practical bearing on anything (even if you display comments in descending order of their rating), it's really for show and nothing else. Of course there's always an exception, and that's 0, which removes the comment from some people's view. So basically, from a practical point of view, there are two types of ratings: zero, and not zero (which includes not rating at all). I fail to see how rating a 1 as opposed to a 4 is "modding down".

[ Parent ]
Ratings == information (3.50 / 4) (#48)
by slaytanic killer on Thu Jan 11, 2001 at 11:03:45 AM EST

In the end, ratings are a form of information. It tells you who rated you, so you have a greater understanding of them. In a sense, it is like getting a response from them why they rated you a certain way -- you can read their previous comments and get a sense of them, why they would object.

But I agree, I get far less information now because people are afraid to rate a 1 for disagreeing. I think I want people to mod me down for disagreeing, so I can learn about them. It conflicts with trusted user status, because I lose every comment that is hidden, but maybe it's better to have the gauge of disagreements.

I don't care (3.40 / 5) (#57)
by mystic on Thu Jan 11, 2001 at 04:23:48 PM EST

I don't care
- who rates me what.
- whether I get trusted user or not.

How much harm from Exposed Ratings? | 77 comments (77 topical, 0 editorial, 0 hidden)
Display: Sort:

kuro5hin.org

[XML]
All trademarks and copyrights on this page are owned by their respective companies. The Rest © 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!