Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

[P]
Subjective ratings? (aka Yet Another Moderation idea)

By Locked in Meta
Wed Mar 07, 2001 at 06:23:53 PM EST
Tags: Kuro5hin.org (all tags)
Kuro5hin.org

I think that kuro5hin should have a rating system that would allow k5ers to rate a comment based simply on whether they agree with it or not.


It would not be a controversial statement, I'm sure, to assert that the k5 rating system (and that of Slashdot and Plastic) is often swayed (polluted? abused?) by the personal biases of those who dish out the karma points. My proposal is an attempt to establish a separation between 'objective' ratings (is that comment relevant to this discussion? If not, then should it be censored? If so, how highly should it be rated?) and 'subjective' ratings (let your hair down and your biases free. Do you agree with what was said? Disagree?). Currently we have one rating system that does the job of two, and I think this should change.

This extra scoring system ("mana"?) would operate on a scale from -3 (disagree) through 0 (eh) to +3 (agree). It wouldn't affect how the comments are displayed on your screen (assuming your k5 preferences are set up like that - the conventional scores would still deal with that); nor would it have the potential to cause a comment to disappear into zero-limbo.

"If someone has something to say, they should reply to the comment."

Maybe people should reply to comments but in practice they often don't. Maybe they don't have the time. Maybe they don't feel they don't have anything to contribute to the thread other than "me too" or "I disagree". Maybe they just can't be arsed. Whatever the reason, and regardless of whether it is right or wrong, sometimes people seek a simple alternative to writing out a reply, and that tends to be rating a post up/down.

Also, I find that it is easier to disagree to a comment/story than it is to agree to one, simply because the former gives you something to refute, and the latter doesn't. Result - "Nodding head syndrome" (or "vocal minority vs contented majority syndrome") and the impression that the original statement is more unpopular than it really is. Providing this new rating system should level the playing field.

"This won't stop moderation abuse."

It won't prevent deliberate abuse but I believe it will reduce the problem of people inadvertently applying a bias to their moderating. There will no longer be any excuse for mixing subjective and objective reviews of a comment. And perhaps it might ease those "I disagree with what he is saying, but I also think that what he is saying is off-topic and completely irrelevant to this discussion. I should moderate him down, but I don't want to appear biased, so..." dilemmas.

"People won't bother voting if it doesn't make any difference to the rating."

I disagree. It will not affect how a comment is displayed (and rightly so), but you wouldn't exactly be casting your vote into /dev/null either. At the very least, people will be able to see how popular/unpopular their opinions are (and who agrees/disagrees with them). The popularity of an opinion may not be an accurate gauge as to whether it is right or not, but nonetheless I think a lot of people would be interested in this data. Plus, the k5 database could correlate these figures and allow you to see who agrees with you the most. Important? No. But try telling me that you wouldn't be curious... :)

In conclusion, this will not completely solve every single karma/moderation related problem, but I reckon it might help.

Sponsors

Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure

Login

Poll
Is this a good idea?
o Definitely! 19%
o Maybe. 22%
o Unsure. 14%
o Not really. 25%
o Definitely not! 18%

Votes: 126
Results | Other Polls

Related Links
o Slashdot
o Kuro5hin
o Also by Locked


Display: Sort:
Subjective ratings? (aka Yet Another Moderation idea) | 61 comments (56 topical, 5 editorial, 0 hidden)
Rating systems are not objective (4.44 / 9) (#1)
by babylago on Sun Mar 04, 2001 at 01:10:43 PM EST

And they're not supposed to be. The whole point of moderation systems is to respond to the subjective interests of the community they serve. Giving a rating a different name and different weight doesn't change anything.

I think you need to distinguish between what is objective and what is a commonly held opinion. If I posted a comment to this that said 'You're just a sleazebag ho', it would more than likely be moderated out, not because it is objectively wrong, but because it is a commonly held opinion that trolls like this do not contribute to the discussion.

So my question in response is: Why do you want to make the moderation process at all objective, when its sole purpose is to subjectively aggregate the interests of the community?

---
[ Blog | Hunnh ]

because (4.00 / 4) (#4)
by Seumas on Sun Mar 04, 2001 at 01:29:33 PM EST

Because, he's probably concerned about instances where perfectly reasonable comments are rated to 0 or 1 for absolutely no understandable reason, other than that the voter apparently has a chip on their shoulder or a stick up their ass. There are examples of this every single day on Kuro5hin and it has become extremely obvious that the voting system is no longer used to seperate the spam from the trolls, flames and content and the trolls and flames from the content, and then the great content from the normal content. Instead, it's used to deter the expression of unpopular or non-party-line opinions. For example, make some comments about abortion or drugs legalization. It doesn't matter which side you voice your opinion on -- someone who very much opposes that opinion will mark you down to a 1 or a 0 -- and their reasons will be obvious after you review the last dozen or two dozen comments that they themselves have posted.

If this continues to become the way a large number of people vote, Kuro5hin will no longer be the place for discussion and debate, but a place where the only opinions that are available to be read and debated by the public will be those that the majority of voters agree with. Hence, if there are more pot-smoking, Amiga-using Ally-McBeal watching Democrats than any other group, only that segment will be readily represented and raised to the top of the discussion.

Yes, it sounds a bit extreme now, but it has to start somewhere, just like trolls and Slashdot started small and insignificicant and grew. The only difference is, at K5, people can have a greater impact than five miniscule votes every week or two.
--
I just read K5 for the articles.
[ Parent ]

Moderation Results = Content (4.80 / 5) (#7)
by babylago on Sun Mar 04, 2001 at 01:46:00 PM EST

You're approaching this as though there were some 'ideal content' out there that k5 is trying to acheive, which is contrary to the notion of moderated systems. So what if some people moderate in a way that you don't like, or if a troll gets posted? Whatever results from moderation by the users of the community IS the content of the system.. Whether or not that contributes to a free and open discussion is not important unless the users, expressing themselves through the mechanism of moderation, say that it is so. If it is prone to abuse, then those same abuse mechanisms are available to you. If there is an opinion with which you disagree, and you are moderated down, does that make your opinion less valuable? That's not really the question. The sum total of your moderation is the sum total of the subjective opinions of people who are interested. Maybe they don't agree with you.

Moderation systems should not be designed as positive reinforcement for self-esteem, but for aggregate evaluation of the value of a particular idea within a particular set of users and circumstances.

Where moderation systems break down is when the content is either at a low volume or is of generally little interest to the community, at which point people start tinkering with the mechanism of moderation systems instead of spending time developing content that will interest the community.

---
[ Blog | Hunnh ]
[ Parent ]

So? (4.66 / 6) (#8)
by Inoshiro on Sun Mar 04, 2001 at 02:02:55 PM EST

uh.. " It doesn't matter which side you voice your opinion on -- someone who very much opposes that opinion will mark you down to a 1 or a 0 "

That is exactly why everyone is allowed to rate. So what if a few people with chips on their shoulders rate wonky? Fuck 'em! 1) it's only numbers on a website (no matter how much I saw this, people like you always get bugs up their asses about it anyways -- but hopefully I'll get through someday), 2) everyone can fucking rate anyways. So rate fairly!

It's all about making people who rate unfairly not count. Over on that other site, the system there has a selected cadre of people who can add or subtract a point once. That's it. For us it's different -- everyone can rate, and everyone can change their rating. It doesn't matter in the end because of how this works.

So if you actually care about ratings (which I still find a bit odd), go ahead and rate fairly. Encourage others to do so, too. That way it'll all balance out.



--
[ イノシロ ]
[ Parent ]
smoothing the wonks (4.00 / 2) (#16)
by iGrrrl on Sun Mar 04, 2001 at 04:29:35 PM EST

So what if a few people with chips on their shoulders rate wonky? Fuck 'em!
But, Inoshiro, I'm a married woman! [bats eyelashes]

In all seriousness, one of the beauties of the system is that the wonky ratings tend to get corrected by non-wonky ratings. Also, enough people rate by quality of the post rather than simply by agreement that most over- or under-ratings tend to even out.

The main reason I've cared at all about ratings (on my more serious replies in threads) is that they might be some level of feedback as to whether I written something coherent and reasonably constructed, or just random wonkiness.

--
You cannot have a reasonable conversation with someone who regards other people as toys to be played with. localroger
remove apostrophe for email.
[ Parent ]

I think you're overlooking something. (3.00 / 1) (#32)
by marlowe on Mon Mar 05, 2001 at 11:26:50 AM EST

Rating someone a 1 is mostly harmless. Rating him a 0 can lead to censorship.

This 0 business is too prone to abuse. It would be better if all users had the option of seeing postings below 1. But even so, I think we can do better.

What we need is a front-end intelligence test, so that people below a certain IQ simply aren't allowed to post. It's possible to construct such a test to be ideology-neutral. Just randomly generated math problems, next-number-in-sequence, and syllogisms involving mythical places, characters and situations. Randomly generated, to foil script kiddies. Everyone who signs up for an account has to take this test once. If he gets above a certain score, he gets an account.

If anyone gets past the IQ test, whatever he posts should be given the benefit of the doubt. No censorship, just selectivity of participants.

The way to get rid of stupid posts is to shut out stupid people. Not people you or I think are stupid. People that *are* stupid, as determined objectively by an impartial server-side script.


-- The Americans are the Jews of the 21st century. Only we won't go as quietly to the gas chambers. --
[ Parent ]
Abuse, feh (5.00 / 1) (#35)
by Inoshiro on Mon Mar 05, 2001 at 05:23:09 PM EST

Any abuse is balanced out by hawk-eyed people like you. A lot of people review hidden comments, thanks to it being in your user bar (we put that in for this reason last October). Rarely do 0'd comments which don't deserve languish in there.

As for a front-end IQ test, things like that are too subjective. The way to get rid of stupid posts is not to censor, it's by way of education. You also have ot understand that some people just don't have the same beliefs/ideas as you do.



--
[ イノシロ ]
[ Parent ]
Almost... (none / 0) (#43)
by Locked on Tue Mar 06, 2001 at 08:25:20 AM EST

Because, he's probably concerned about instances where perfectly reasonable comments are rated to 0 or 1 for absolutely no understandable reason, other than that the voter apparently has a chip on their shoulder or a stick up their ass.

I suspect that most people would blanch at the idea of rating down a comment they disagree with, but see nothing wrong with rating up a comment they agree with.


Locked

[ Parent ]

Bingo (5.00 / 1) (#27)
by kmself on Mon Mar 05, 2001 at 05:39:13 AM EST

You've pretty much nailed it.

K5 moderation is an aggregate measure. The primary objective is to accurately reflect the data in the moderation system. A related goal is to have the system tend to seperate wheat from chaff, but this is an ideal, not a realistic goal. K5's moderation comes close, and avoids several inherent problems in Slashdot's moderation (slow responsiveness, many false negatives).

There are a few tweaks I'd like to see made to Scoop, largely an ability to report and/or filter by standard deviation (a measure of controversiality of comments), and to run some more complex (and/or ad hoc) stats on comments and moderation to ferret out abusers. But by and large it works.

One of the things you can't do to solve the problem of people providing bad inputs to a system is to address the problem by getting more inputs. In all likelihood, you're just going to get more bad inputs.

I'm one of the people who provided much of the input and rationale for the K5 moderation system. And, no, I'm not always happy with how my own posts (particularly editorial comments) get moderated. But, then, that's a feature, isn't it ;-)

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

A new, better troll. (3.80 / 10) (#3)
by tiamat on Sun Mar 04, 2001 at 01:15:49 PM EST

Hum, this new system could give the trolls something useful to do. Instead of just being plain offensive and obviously trollish, they could aim to get the (5,-3) score for their comment. (A perfectly good comment, that EVERYONE disagrees with.)

In reality though, would it be possible for anyone to get a 5 score for normal moderation, while getting a perfect -3 subjectively? [I realise it's possible in theory, but would anyone ever vote that way?]

And instead of destroying someone's trusted user status by rating all their comments 1 or 0, you could instead rate them all -3 subjectively and destroy their self esteem. <grin>

Seriously though, this would be a large change to make in the k5 community. Right now only the people who actually make comments have a say in how much they agree or disagree with you. (That's assuming that the people who moderate do it well, and/or you can't determine the biases of bad moderators very well.) On the other hand, with this new system everyone can effect the conversation by having a passive vote on whether or not they agree.

So do we as a community WANT the people who don't comment to have this vote? I for one don't know.

the big problem (4.25 / 8) (#5)
by Seumas on Sun Mar 04, 2001 at 01:35:26 PM EST

The big problem is that there's a point where the amount of moderation to be done on a particular post isn't worth the energy. It's enough as it is to bother to vote on each comment you read, let alone vote on each comment twice, with different intentions and ratings in mind for each. There is such a thing as influence overkill. In fact, this is a reason why I would be all for having only three votes on any item. A 0 (spam) a 1 (flames, mild trolls, normal content) and a 2 (great content). Or, even just a 0 and a 1 to moderate crap down and leave everything else at an even pace. I don't need all of my comments to be rated at 4's and 5's -- I'll settle for just being viewable as much and no more than any other non-spam/troll post.
--
I just read K5 for the articles.
could fix this by... (none / 0) (#12)
by xriso on Sun Mar 04, 2001 at 02:28:16 PM EST

... making it so your ratings override others'. If I vote a comment as 5, I will see it at the top regardless of what other people rate it. This could increase motivation to moderate.
--
*** Quits: xriso:#kuro5hin (Forever)
[ Parent ]
Don't label me an abuser, just because I disagree (3.77 / 9) (#9)
by Eloquence on Sun Mar 04, 2001 at 02:04:36 PM EST

As I've pointed out before, I rate comments based on whether I agree or disagree with them (ironically, some people have moderated me down to 1 for saying this). What is the alternative to this? You write:

'objective' ratings (is that comment relevant to this discussion? If not, then should it be censored? If so, how highly should it be rated?)

Whereas you don't define what these "objective" standards for rating a comment highly should be. The only one I have seen mentioned here is how well-worded a reply is. I'm not interested in reading a however-well-written pro-tobacco diatribe, the ramblings of a creationist or the opinions of a neo-nazi. If I've come to the conclusion that these people are wrong, then I don't want to read what they have to say, unless it's some really new and brilliant argument that I have never heard before and that might alter my view on the subject. And if I think they're wrong, I don't want others to be exposed to their views either.

If I just want to read something well-written without no content that interests me, I might as well fire up the postmodernism generator and masturbate mentally to its random-generated results, complete with random-generated sources.

When I rate comments, I want to rate the ones that stand out, the ones that give me new information or that might give others new information on a subject, those that make a logical argument that I, personally, find convincing. All these judgments are subjective. Much of what I have written has been rated 5 by half of the people and 1 by the other half. So what? That's how it's supposed to work. That is no abuse. If you label me a "rating abuser" because I use the system how it is intended, well, then I can return the name-calling and label you an abuser, because you push comments to the top that have no merit whatsoever or are even dangerous (Jews control the media, Auschwitz never happened, the government poisons the foodsupplies..) -- just because they are more eloquent than others. Likewise, I could call it abuse to rate a comment down that includes just a perfectly relevant link or a short argument that perfectly sums up the debate on a subject. In fact, the shorter and more to the point, the better -- this is an ability that I myself often lack and admire in others.

Now, as for your suggestion to separate "opinion" and "fact", this only makes sense for the negative ratings -- I can see a more Slashdot-like moderation as a variant of the current system, one where the positive posts are selected, as compared to trying to find an overall value for every post. Contrary to Slashdot, however, the maximum value shouldn't be +5, it could be open-ended. Negative ratings should only be used in such a system for spam or clear insults. However, contrary to Slashdot, don't show hidden comments to non-trusted users; this would only create a community feeling for those who post them. I don't believe that K5 will make such radical changes to its moderation systems, however.
--
Copyright law is bad: infoAnarchy Pleasure is good: Origins of Violence
spread the word!

So what do you do when an article... (3.00 / 1) (#31)
by marlowe on Mon Mar 05, 2001 at 11:15:44 AM EST

that you disagree with gives you new information? Which principle has higher priority?

Be honest now. And this bit about a "new and brilliant argument" is fishy. Who decides what's brilliant? Surely not you. That would be a conflict of (ego) interest. And why should the validity of an argument depend on its age?


-- The Americans are the Jews of the 21st century. Only we won't go as quietly to the gas chambers. --
[ Parent ]
Agree vs. Disagree (5.00 / 2) (#34)
by Eloquence on Mon Mar 05, 2001 at 03:35:23 PM EST

[when an article] that you disagree with gives you new information?

I try to evaluate whether the information is true or false. If I cannot say so at the point I am reading the comment/story, I refrain from rating it. If it is true and relevant, I will change my view on the subject.

Who decides what's brilliant? Surely not you.

Of course I do! That's the whole point of the rating system -- generating a somewhat objective value by combining as many subjective ones as possible. It's a system of peer review. What do you think is done when you submit a scientific paper? They don't judge it based on whether it is especially eloquent or not (at least not in the natural sciences), they judge it based on whether they agree with it, whether its logic makes sense to them.

But I see how your strange opinion on ratings could be explained. You probably see agreement as something that is merely emotional and has nothing to do with facts. But for me, it is exactly the opposite. When an agreement or disagreement is merely a "gut feeling" ("I don't want this to be true") but I have no knowledge on the subject, of course the last thing I would do is rate someone down for saying it. My agreement or disagreement as expressed by my ratings is always based on my knowledge base.

My post above currently has a rating of 3.83. You have rated it 2, two others have rated it 5 -- I presume that those who rated it 5 found it quite convincing and logical, and that you, judging from your reply, didn't, in other words you voted me down because you disagreed with me. That's the way it's supposed to work (although it's seriously schizophrenic in your case ;-). The idea is that if an argument is really logical and convincing, it will automatically rise to the top, while false statements will drop to the bottom.

What really annoys me is seeing perfectly relevant arguments and questions moderated down just because they are short. Especially questions are often moderated down, as if admitting that something was not understood was somehow of no value to the discussion. That is really not good for discussions. And generally, as I've pointed out before, I would prefer a moderation system that is more focused on rewarding good posts than on rewarding good ones and punishing bad ones.
--
Copyright law is bad: infoAnarchy Pleasure is good: Origins of Violence
spread the word!
[ Parent ]

Ain't nobody here but us non-emotional disagreers. (none / 0) (#44)
by marlowe on Tue Mar 06, 2001 at 12:18:45 PM EST

We're all perfectly rational and objective. Yes, we're sure. Take our word for it. And even if we were subjective, all our subjective opinions would add to an objective judgement somehow.


-- The Americans are the Jews of the 21st century. Only we won't go as quietly to the gas chambers. --
[ Parent ]
I agree with you entirely... (none / 0) (#48)
by Paul Crowley on Wed Mar 07, 2001 at 06:55:18 PM EST

... so, of course, I've rated you up!
--
Paul Crowley aka ciphergoth. Crypto and sex politics. Diary.
[ Parent ]
Irony? (none / 0) (#52)
by zek93 on Thu Mar 08, 2001 at 05:19:02 PM EST

As I've pointed out before, I rate comments based on whether I agree or disagree with them (ironically, some people have moderated me down to 1 for saying this).
What if they're rating you at 1 in order to keep you from getting trusted user status and applying zero ratings to comments that you *really* disagree with? Would you still consider that to be an example of irony?

Now if you'll excuse me, I'll go give your last 30 comments a rating of 1, just for that purpose. Go ahead and retaliate if you wish -- I don't care about the Mojo level of this particular account.

[ Parent ]

Cute (none / 0) (#53)
by Eloquence on Thu Mar 08, 2001 at 10:17:57 PM EST

Ironic? No, just pathetic.
--
Copyright law is bad: infoAnarchy Pleasure is good: Origins of Violence
spread the word!
[ Parent ]
Pathety? (none / 0) (#55)
by zek93 on Fri Mar 09, 2001 at 08:14:04 PM EST

Ironic? No, just pathetic.
That's interesting. Do you also consider your comment rating policy to be "pathetic"? In a sense, my rating of your comments to 1 is a logical extension of your rating policy. On one hand, you are influencing the rating of a post based on whether you agree with it, and on the other hand, I am influencing your mojo based on whether I agree with your rating policy. While you find it important to keep messages you disagree with at the bottom of a ranked list, I find it important to keep you from getting trusted user status. Each of us is basing his actions upon his opinions, working for some goal.

[ Parent ]
No, pathos. (none / 0) (#56)
by Eloquence on Sat Mar 10, 2001 at 10:46:41 AM EST

In a sense, my rating of your comments to 1 is a logical extension of your rating policy.

Even if your judgment of the potential that I may use the zero-rating other than it is intended is right (although I say that I don't), your actions make no sense. Some readers will find it annoying that you are pushing posts to the bottom of their view (sorted by rating) that they would have liked to read first. If you continue your "preventive 1-rating" on all people who, according to your judgment, might act unfairly in the future, the resulting frustration by many others will further deteriorate discussion and lead to unnecessary meta-debate. If your judgment is wrong, you are an even more annoying, pathetic troll. As you could see by examining the past zero ratings, I have not abused the zero rating in the way you suggest (and I totally agree that doing so would be an abuse), so your future extrapolation that I might do so has little basis in reality.

Thus, you are, again, an annoying, pathetic troll. People like you will be responsible for ultimately breaking the current moderation system. You are just a more rationalized version of the Slashdot spammer who justifies his actions by blaming them on the unjustness of Taco's story selection. The use of multiple accounts to commit actions that would be seen as wrong by most other users is also typical for the Slashdot trollfolk. So, congratulations, you may be among the first real K5 trolls.
--
Copyright law is bad: infoAnarchy Pleasure is good: Origins of Violence
spread the word!
[ Parent ]

Heh (none / 0) (#59)
by zek93 on Mon Mar 12, 2001 at 03:31:57 PM EST

Heh, I suppose I could declare myself the "winner" of this debate since you've resorted to name-calling while I haven't, but that's not my style.
Some readers will find it annoying that you are pushing posts to the bottom of their view (sorted by rating) that they would have liked to read first. If you continue your "preventive 1-rating" on all people who, according to your judgment, might act unfairly in the future, the resulting frustration by many others will further deteriorate discussion and lead to unnecessary meta-debate.
How is this different from your behavior of rating down comments that you disagree with? I don't see why people that would get annoyed at such a thing wouldn't be any less annoyed when the down-rating is done by you. In any case, your actions have already spawned meta-debate. Today I saw a complaint by someone (I think it may have been theboz) about your ratings tactics.
As you could see by examining the past zero ratings, I have not abused the zero rating in the way you suggest (and I totally agree that doing so would be an abuse), so your future extrapolation that I might do so has little basis in reality.
I do not currently have trusted user status.
People like you will be responsible for ultimately breaking the current moderation system.
Some (like myself) would say that your actions (rating based on agreement) are doing more to ruin the system than my attempts to lower your mojo. You may not see this, as you apparently think that rating-on-agreement is a good policy for some reason, but I assure you that there are consistent value systems that consider your strategy a contemptible abuse of the system.
So, congratulations, you may be among the first real K5 trolls.
I am not a troll. I am, at worst, a ratings abuser. If you think that I am among the first of those here, then you haven't been paying attention. The question of whether or not I am a ratings abuser is a matter of debate. Obviously, each of us thinks of the other as a ratings abuser, and of ourselves as not-an-abuser. Of course it is natural human behavior to think of one's own actions as just, and opposed actions as unjust.

[ Parent ]
Just out of curiousity... (4.40 / 5) (#10)
by br284 on Sun Mar 04, 2001 at 02:12:13 PM EST

Just out of idle wondering, but how many people actually go out of their way to rank comments? I for one, do not unless I find something that I believe is extra-deserving of attention.

I figure that the S/N ratio here at k5 is quite acceptable, and for me, ranking comments really doesn't do a thing for me as I read through the discussions with all of them showing up. (I guess the analogy would be reading with /. at the -1 threshold.)

And as far as moderation abuse happening at k5, I don't think that I've seen any cases of it happening. Granted, I do not have a trusted user status, and do not follow or care who else does. Is moderation abuse really a problem here? If so, can someone give me some examples, as I would be very interested in hearing about them.

I guess what I'm saying is the old saying, "If it ain't broke, don't fix it." k5 is far from broken (from my perspective), so why can't we just let it run as it is and if a problem pops up, worry about fixing it then?

-Chris

I rank comments in a fraction of the stories (3.00 / 1) (#21)
by roystgnr on Sun Mar 04, 2001 at 11:28:22 PM EST

My behavior is quite simple:

If I'm busy or only vaguely interested in a story, I skim down a long story's comments section and only read the comments rated 4+. Same thing I try to do on Slashdot, but frankly the filtering is much better here.

Normally, I'll read the 4+ comments, skim every comment, and only moderate the comments that appear seriously misscored. I wish I could give you examples, but it would take me a while to hunt.

If I'm bored, or extremely interested in a story, I read and rank every comment. I honestly do think that the rankings make more than an ego difference, because I use the rankings. The reason why Kuro5hin is so much better than Usenet is that, when wading through a thousand posts in each place, it's so much easier here to find the top ten/hundred/five hundred that you have time to read.

It's not a bad system; as long as many other people behave the same way (and our "busy/bored" cycles don't correlate too closely) it keeps signal-to-noise nice and high with a minimum of work.

[ Parent ]

I rank comments in a fraction of the stories (none / 0) (#29)
by mami on Mon Mar 05, 2001 at 10:52:46 AM EST

I stopped ranking at all and do not care for it either in selecting what I read. I really don't know why I should trust other people's rankings and not make my own judgement. Aside from filtering out obvious flamebaits, spams and trolls or pointing out highly informative technical comments, I could live without any ratings.



[ Parent ]
I rank if I think it needs adjustment (none / 0) (#60)
by Pseudonym on Wed Mar 21, 2001 at 07:34:51 AM EST

"If it ain't broke don't fix it." Couldn't have said it better myself. I rank stories if I think they're not at the right level. For example, if they're unranked or only one or two people have ranked it and I think they're on the wrong track (e.g. if I think they've ranked it based on how much they agree rather than how much it contributes), I'll put in my entry.

Oh, and I often rank replies to my own comments. Dunno why. Probably because I see them first on my personal "comments" page.


sub f{($f)=@_;print"$f(q{$f});";}f(q{sub f{($f)=@_;print"$f(q{$f});";}f});
[ Parent ]
Huh? What?!? (3.50 / 10) (#11)
by Carnage4Life on Sun Mar 04, 2001 at 02:16:42 PM EST

I think that kuro5hin should have a rating system that would allow k5ers to rate a comment based simply on whether they agree with it or not.

Why? You do not state a single benefit of this system over the current one.

Basically it seems all you want is to devolve kuro5hin discussions into AOLer style "Me Too" campaigns without having to risk posting an actual "Me Too" and get moderated down for it.



Simple rules good, complex rules bad (4.75 / 8) (#13)
by itsbruce on Sun Mar 04, 2001 at 02:33:26 PM EST

You can't legislate for a healthy community. The more you try to do that, the more you defeat your purpose as honest expression and individuality is tied in knots and weighed down by the rules. IMO any proposed rule that adds complexity without clear benefit is a bad thing.

So I rate this suggestion a bad thing.


--

It is impolite to tell a man who is carrying you on his shoulders that his head smells.
this story's score is indicative (3.50 / 10) (#14)
by el_guapo on Sun Mar 04, 2001 at 02:59:08 PM EST

of our moderation problems, methinks: don't vote against this story if you disagree with the idea damnit!!!! i mean, doesn't that just sort of validate the author's points???? k5 moderation, as fucked up as /.'s, i'm afraid (mod me down, please - i'm sick of being trusted and seeing the stuff modded a zero simply because people disagree with it)
mas cerveza, por favor mirrors, manifestos, etc.
Just stick with what we got. (3.28 / 7) (#15)
by DeadBaby on Sun Mar 04, 2001 at 04:02:56 PM EST

I'm sick of new moderation systems every few days. Just deal with what we have.
"Our planet is a lonely speck in the great enveloping cosmic dark. In our obscurity -- in all this vastness -- there is no hint that help will come from elsewhere to save us from ourselves. It is up to us." - Carl Sagan
Discussion vs. Consensus (4.25 / 4) (#17)
by HypoLuxa on Sun Mar 04, 2001 at 06:10:25 PM EST

I think that one of the big points on k5 is have discussion. In order for that to happen, we have to have opposing viewpoints. The problem I see with a moderation system such as this is that it really asserts what the "right" and "wrong" opinions are, and that kind of value judgement is the antithesis of good discussion. If you start assigning that kind of number to each post or article, then this is nothing but geek culture dogma, and looses so much of it's value.

--
I'm guided by the beauty of our weapons.
- Leonard Cohen
Question for all those for this idea... (4.78 / 14) (#18)
by Carnage4Life on Sun Mar 04, 2001 at 06:31:32 PM EST

I've noticed a few people seem to be for this idea so I have a quick question for them:
Besides promoting groupthink, what can this proposed moderation system achieve?
The main reason several people here are refugees from Slashdot (myself included) was the fact that the moderation system became a way to reward people for spouting the party line and punish those who dissented. It would be extremely saddening for kuro5hin to go even further down that road by explicitly creating a moderation system with that as the goal.

Currently on kuro5hin I've seen well written posts that disagreed with the general tide of the article and comments get highly moderated primarily because people try to rate objectively. What would be the purpose of having a post given a rating and then being given a score for how many people didn't agree with it? Isn't it more likely that the people who dissent will vote as opposed to the people who agree especially of the people who agree have already given it a good objective score? This will simply skew the subjective score towards the dissenters.

A subjective grade simply encourages Trolls and Karma Whores to come out of the wood work as Slashdot's moderation system has done. I hope it is never implemented here.



I see at least one good use... (3.50 / 2) (#24)
by pb on Mon Mar 05, 2001 at 02:57:11 AM EST

Those who wish to rate comments subjectively can do so. Then, if *you* can choose to view one rating system or the other, you can do so.

If this actually led to better objective ratings, I'd be all for it. And I'd probably view both ratings, and sort by objective ratings. Some people might choose to only rate subjectively, or maybe rate as both.

Anyhow, I think it's an idea that has potential, and needs some fleshing out. But just telling people to read the FAQ doesn't seem to cut it around here anymore.

And no, I don't think adding this feature would lead to The End Of Kuro5hin(tm). People already DO rate posts subjectively, and I'd like to be able to filter those ratings out. Instead, I end up reading a lot of posts...
---
"See what the drooling, ravening, flesh-eating hordes^W^W^W^WKuro5hin.org readers have to say."
-- pwhysall
[ Parent ]
Feedback to the author (none / 0) (#25)
by infraoctarine on Mon Mar 05, 2001 at 03:39:37 AM EST

Besides promoting groupthink, what can this proposed moderation system achieve?

As I've been thinking along these lines before, I know why I would like it. As a simple way to give/receive feedback on the post. Sometimes you'd just like to let the author know that you liked his/her message, but posting a message saying "Yay, me too man" is just pollution of the thread. And from the authors view, it would be interresting to see what others think about your ideas/views; it would be like a built-in poll in every post.

The idea might need some more work, but I think it has its merits.

[ Parent ]
Why it would _reduce_ groupthink... (none / 0) (#41)
by Locked on Tue Mar 06, 2001 at 07:30:38 AM EST

It would actually reduce groupthink, by harmlessly diverting the 'groupthink bias' into the subjective rating systems. Sure, some posts will be more popular than others, but that won't affect how the comment is displayed, or the author's mojo.

The efforts of 'karma whoring' would be almost rendered useless. Potential karma whores can apply all the sophistry and 'Please don't flame me for saying this...' pleas they want, but it won't increase their mojo. Their comments will be judged on their merits, not their appeals to the gallery.


Locked

[ Parent ]

Yes! (3.00 / 5) (#19)
by bjrubble on Sun Mar 04, 2001 at 06:46:24 PM EST

I think it's naive to tell people to just ignore their personal reactions to a comment when rating. Two datapoints, two axes. Makes perfect sense.

I'd like to see this extended beyond just a numerical score, though. I'd be more interested in the distribution of the ratings. If one item has an aggregate rating of 0 because everybody found it boring, that's different from something that divides the community and gets pulled equally by both extremes.

Why agree/disagree comment rating is broken (3.66 / 3) (#23)
by Sunir on Mon Mar 05, 2001 at 02:02:37 AM EST

The whole concept of comment-by-comment rating is flawed. Ultimately, all you need in response to an article is the essence of the argument. If you disagree with something and the rebuttal isn't present, you'd say your piece. If your views are already represented, there would be no need in adding anything. It would just redundant.

The reason why we have issues with this is that we can't do this here. First, we can't consolidate redundant information. Second, we can't agree with one paragraph of a post and disagree with the second through the rating system. We can only agree or disagree with the whole thing. Third, because we have to add a lot more text in replies to compensate for this coarse granularity, people don't read the hundreds of comments before replying. Fourth, it's hard even if they wanted to because the salient points may be buried seven posts deep in a long thread.

Of course, my opinions are biased because I run the other other site that does this (as it is a wiki). But none of MeatballWiki's functionality exists here. What we want are practical solutions for the environment here.

First, we can all stop posting long-winded essays (like this post), and instead break our comments into fragments that can individually be raised, lowered, or replied to. Second, we can read the whole set of comments before we post. Third, we can try doing an experiment collaborative editing for a change. That is, if you would like to change another post, copy and paste it, make the changes, then post the new text.

I don't seriously think we could do any of the above. But it's fun to think about it.

"Look! You're free! Go, and be free!" and everyone hated it for that. --r

Did I really do that (2.50 / 4) (#26)
by FeersumAsura on Mon Mar 05, 2001 at 04:43:29 AM EST

As I didn't like your idea and was worried about people agreeing with you I voted -1 to help remove it from the submission queue. Trying to make it so that other people wouldn't see it and agree with you. It just seems wrong doesn't it?

I'm so pre-emptive I'd nuke America to save time.
Bias is a design feature (4.87 / 8) (#28)
by kmself on Mon Mar 05, 2001 at 06:09:39 AM EST

I'm one of the people who's had more input on the moderation system than most. I've previously posted a moderation backgrounder you may find useful.

While I appreciate your comments, I believe your suggestion is flawed in several ways.

  • You fail to indicate how an additional measure might be used. Ultimately a thread sorting or ordering algorithm needs to be reduced to a unidimensional measure. The current moderation system is really asking "where on a linear scale would you prefer to see this item ranked?". Moderation + mana still gets reduced to a single value ultimately.
  • If the fundamental problem is user bias, adding more opportunities for the same users to bias the system, without checks, corrections, or countermeasures for bias, simply creates a more confusing system without benefit.
  • The data are already in the system. An article which is (generally) good or bad will tend to have a high or low value, with little deviation among moderations. A highly controversial comment will have a higher deviation. The statistic of interest, standard deviation, can be computed readily, and would be a good candidate for inclusion in a content filter at K5. It's not currently included in Scoop, that's nothing but an implementation issue.

Read my notes and/or search for various discussions of the moderation system at K5 and the Scoop discussion sites. Most of your ideas have been presented and rejected previously.

And, frankly, I've got no problems with an article in which a single, evident truth is posted in response, generating nothing but moderations to that one post. Sometimes the truth is simple.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.

I disagree (none / 0) (#51)
by bjrubble on Thu Mar 08, 2001 at 05:26:13 AM EST

It terms of thread sorting, you yourself point out one useful metric: standard deviation. As for the bias, I'd view it more as a bias sink -- it's a place to indicate explicitly how you feel about a post, relieving the temptation to use the technical ratings for editorializing.

[ Parent ]
Bias sink? (none / 0) (#54)
by kmself on Fri Mar 09, 2001 at 07:30:39 PM EST

Sorry, I'm afraid I'm unfamiliar with the term or your intent by it. Could you explain more clearly?

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Bias sink (3.00 / 1) (#57)
by bjrubble on Sun Mar 11, 2001 at 06:10:04 AM EST

A bias sink is a place to express your bias other than the rating. I think one reason people don't give good ratings to posts they disagree with is that it's extremely unsatisfying. If you give them the opportunity to scrawl "I think it stinks!" on the side, they may be more fair when rating it.

[ Parent ]
But you've got that... (none / 0) (#58)
by kmself on Sun Mar 11, 2001 at 06:40:41 PM EST

It's the reply-to field. You can rate a comment (or abstain from rating), and/or respond to it. The rating encapsulates all. The comment is as free-form as you can get (well, within the limits of text and HTML).

Speaking for myself, a very well articulated response, though I disagree with it, will be rated higher on account of its presentation. A poorly written response I do agree with I'll often mark down. I do my own weight juggling, and would expect others to do similar.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

If you're just gonna approve your own opinion... (3.00 / 4) (#30)
by marlowe on Mon Mar 05, 2001 at 11:05:46 AM EST

why bother with a public weblog? Make a program that runs on your own PC and spits your own opinions back you, with random names stuck on them, to create the illusion that there are peopel out there who all completely agree with you. You don't even have to dial up your ISP to use it.

Call it Solipsism 1.0

-- The Americans are the Jews of the 21st century. Only we won't go as quietly to the gas chambers. --
Not sure of its relevance ... (3.66 / 3) (#33)
by loaf on Mon Mar 05, 2001 at 01:42:20 PM EST

But I just noticed that you can now see the IDs of those who voted +1 in the moderation queue.

This seems to me to related to the topic, see what else they voted +1 for ... and maybe the next evolution would be to allow you to tinker with the weights of others, so that their input would be valued more highly when you are filtering on comment score.

E.G. You voted +1, you scan down the list and you see that I voted +1. You follow the link and see that I seem like the kind of guy who says things you agree with. You add a little weighting factor and the next time you read a story, my rating of 4 for a comment is now worth 4 + 50%.

How 'bout it Rusty? :)


Do you actually look at the rating of a comment? (4.25 / 4) (#36)
by GreenCrackBaby on Mon Mar 05, 2001 at 05:55:25 PM EST

Find me a k5 reader who actually pays attention to the rating of a comment. There is such low traffic here (which is nice) that I can easily browse all the comments quickly. On a site like /. it may make a difference, but unless you read every comment and don't go by karma points, you're stuck reading only those posts that the majority of people agree with. This creates, in my view, a very one-sided debate.



You're not the first one to ask this. (none / 0) (#37)
by ZanThrax on Mon Mar 05, 2001 at 07:08:54 PM EST

So rather than repeat myself needlesly, I'll just suggest you read this If a topic only has a couple dozen comments, I probably do read them all, but on threads that have a hundred or more by the time I get to it, I probably don't feel like spending an hour or three reading all the lower quality posts, and I'm happy to be able to read the high rated ones first. Basically what I'm saying is, don't assume that everyone reads kur5hin the same way you do.

Before flying off the handle over the suggestion that your a cocksucker, be sure that you do not, in fact, have a cock in your mouth.
[ Parent ]

So what you are saying is... (none / 0) (#39)
by GreenCrackBaby on Tue Mar 06, 2001 at 01:28:26 AM EST

...that in a story with high volumes, you'll only read the comments that the majority of people who read k5 agree with?

I've hardly ever seen real good comments that are very anti-k5ish (i.e. a comment defending riaa) that get high ratings. People should be rating comments based upon what they contribute to the discussion -- NOT based upon how much they agree with the statement.

[ Parent ]

NO (none / 0) (#49)
by ZanThrax on Wed Mar 07, 2001 at 10:03:34 PM EST

I'm saying that in a high volume story, I'll sometimes only read the comments that the majority of people who read k5 think are well written.

I've often seen well written comments that I disagree with vehemently rated high. Just because some people rate based solely on what they think of the idea, there are at least as many who rate solely on the writing. Most people (myself included) consider both factors. I'll rate something thats well written, but with flawed logic at 3. Well written flamebait is still flamebait, but a well-expressed comment from trhurler that makes me think about his position is worth a 5.

In a big thread, the stuff floating between 1 and 2 and a half or so is mostly crappy one liners, usually including flamebait or out and out hatred. (although sometimes they're just meept or Yeah! Right on comments)

Before flying off the handle over the suggestion that your a cocksucker, be sure that you do not, in fact, have a cock in your mouth.
[ Parent ]

Kuro5hin comment moderation doesn't mean much (none / 0) (#46)
by RandomPeon on Tue Mar 06, 2001 at 09:29:13 PM EST

Unfortunately, the moderation on Slashdot is generally more informative than on Kuro5hin. First, positive moderation in slashdot has value - it's rare, so it's valuable. Every post on Kuro5hin could theoretically receive a score of 5, and sometimes it seems close to that - not that this is bad, just that it happens.

I agree that personal prejudices may play into the moderator's decisions, especially when they have strong opinions on the subject. But I don't think people are conciously vindictive - for the most part we don't mod comments we disagree with less higly on purpose, instead we mod them less highly because we generally think we're right. Giving imperfect humans an opportunity to admit their failings isn't gonna help things - we're too proud to admit we have biases.

You see this problem in the "real world" too. Publications which are supposed to be objective get reputations as "liberal" or "conservative". Despite all the journalism school indoctrination to be objective, editors probably run editorials they agree with more often because they want to run smart editorials, and when has a columnist from the other side of the political spectrum ever said anything smart? (My personal example: I expect George F Will to say something profound and brilliant 1-2 times/year. The rest of the time I expect him to regurgitate right-wing crap. But he's actually pretty smart.)

[ Parent ]
Nicely Played! (none / 0) (#50)
by pallex on Thu Mar 08, 2001 at 04:56:23 AM EST

The only reason i give a s*** about moderation is to prevent the sort of crap you see on `the other site`. That doesnt seem to be a problem here (dont know why, i really havent looked too closely into exactly how this site is moderated, and rarely mod anything myself, unless its worth a 5), but who cares - as long as the grits and goats etc are kept away, i`m happy to skim read the lot (if i`m interested). My only criticism of this site is that there are far too many `the world sucks, things would be different if i were in charge` type stories. Then again, when you`re in the mood, they`re a giggle!

[ Parent ]
Filtering and meme death (2.50 / 2) (#38)
by cameldrv on Mon Mar 05, 2001 at 09:25:43 PM EST

The problem with collaborative filtering type systems is that you end up seeing comments you agree with. Since K5 is a discussion site, you build your profile of ratings, and then you just see the comments you agree with. Only seeing what you already agree with doesn't make for very interesting discussion. I come to K5 to hear NEW things that other people have to say about timely topics. If these conflict with my own, yet a good argument is presented, I have to adjust my worldview. I like doing this. However, many others don't like adjusting their worldview, and instead take pleasure in having theirs reinforced.

The problem K5 has been having, as I see it, is that too many of the latter type have been arriving and moderating. More important even than comment moderation, however is story moderation. I have seen a number of stories which were original and interesting be voted down, while the familiar linux/copyright/rights violation type stuff gets immediately modded up. This problem may well be intractable and there will have to be a new exodus. I'm sure that this is already happening as I have seen the originality and quality of stories and comments go way down in the past six months.

I have a couple of other sites I've been looking at for the next cool discussion site, none of which I will ever post to K5, so as to try and delay their disintigration. Face it -- in any cool new movement, the commnunity degrades as time goes on. I guess it's just a simple measure of open-mindedness that one has his feelers out for the next cool site. Those that do get there first, and those are the people who say interesting things. Generally those that get there later aren't as interesting. It's too bad that it has to be like this, but you see it going on all the time with bands, books, theories, and virtually every other type of meme. Usually the people who catch the meme first are the people who catch the most memes, and thus have the broadest range of experience. Therefore they generally are more insightful and interesting.

Or just have more free time... (none / 0) (#40)
by inpHilltr8r on Tue Mar 06, 2001 at 03:53:36 AM EST

So, in essence, what you're saying is, "My UID is lower than yours, nyaahh nyaaah nyaaaah!"


[ Parent ]
Not about that (none / 0) (#42)
by Locked on Tue Mar 06, 2001 at 08:24:19 AM EST

The problem with collaborative filtering type systems is that you end up seeing comments you agree with. Since K5 is a discussion site, you build your profile of ratings, and then you just see the comments you agree with. Only seeing what you already agree with doesn't make for very interesting discussion. I come to K5 to hear NEW things that other people have to say about timely topics. If these conflict with my own, yet a good argument is presented, I have to adjust my worldview. I like doing this. However, many others don't like adjusting their worldview, and instead take pleasure in having theirs reinforced.

That isn't the intention of this idea (in fact, I had never even thought about that). In fact, it would probably be impossible. 'Only show comments from authors I agree with' would be more feasible. But like I said, this isn't what this is about.


Locked

[ Parent ]

interesting idea, however... (4.50 / 2) (#45)
by skeezix on Tue Mar 06, 2001 at 01:27:49 PM EST

I don't think this would be terribly useful for several reasons. First of all, the real place to state your opinions are in the reply to the comment. Secondly, a great many comments don't fit in the form: agree or disagree. What about a factual comment, or a link giving information, or a correction.

Only two measures? More! More! (2.00 / 1) (#61)
by Pseudonym on Wed Mar 21, 2001 at 07:40:06 AM EST

After that, we need another measure for how funny we find it, and another one for how much we sympathise with the poster, whether we agree or disagree with what they have to say, and another one for if we think it's not strictly relevant to the discussion but it's still an interesting sidetrack... we could be here all week.

Quite frankly, I can tell what people think of a post by the subject lines in the followup. People usually make them meaningful. In this thread alone, we have "I agree", "Bingo", "because", "Almost...", "So?" and "So what you're saying is..." and those are only the generic ones.


sub f{($f)=@_;print"$f(q{$f});";}f(q{sub f{($f)=@_;print"$f(q{$f});";}f});
Subjective ratings? (aka Yet Another Moderation idea) | 61 comments (56 topical, 5 editorial, 0 hidden)
Display: Sort:

kuro5hin.org

[XML]
All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!