Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

[P]
Abuse of Comment Ratings

By antizeus in Meta
Wed Dec 13, 2000 at 08:38:27 AM EST
Tags: Kuro5hin.org (all tags)
Kuro5hin.org

Lately the comment rating system has seen some abuse, wherein at least one individual has targetted others and given all of their comments a rating of 1. There are surely worse problems in the world today, but it is somewhat annoying. What can be done to fix this problem?


I and some others (for example, enternone and Beorn) have apparently become the targets of one or more "coward raters", who will rate every comment by a particular individual at 1.

It has been pointed out (see Dacta's comment in the Beorn diary for a compelling argument) that comment ratings do not amount to much, and that we shouldn't really care about this behavior. This makes sense, but part of me is still annoyed. Yes, it's stupid, but humans aren't completely rational. I can easily imagine someone more sensitive than myself taking greater offense, and abandoning the kuro5hin "community" because of it. Even if the hostile ratings themselves do not directly cause someone to leave, it might be a contributing factor.

Some ideas for fixing this problem have been proposed (see rusty's comment, also in the Beorn diary), one of which is the idea that ratings not be considered "real" until a certain number of users apply a rating. This might work. It might not. If this threshold is some static number, then a coward rater need only create that many accounts and run a rating script to achieve his desired effect. If the threshold is a percentage of the number of users, then we may very well end up with a situation in which none of the comments are rated because not enough users have bothered to use the rating system. Personally, I find the idea of rating every comment that I read (or even a significant number of comments) to be tiresome, so I don't bother. It's not like voting on articles, where you have a smaller number of more significant choices to make. I'm sure many others feel the same way.

In my opinion, comment ratings are a failed experiment. Thanks to the coward raters, we can't trust the ratings (as they currently stand) to provide a meaningful measure of the value of a comment. I'm also pessimistic about the ability to fix the system. Perhaps it's time to do away with comment ratings altogether.

Sponsors

Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure

Login

Poll
What's the solution?
o Keep the system as it is. 34%
o Get rid of the comment ratings. 13%
o Tweak the system somehow. 51%

Votes: 52
Results | Other Polls

Related Links
o Kuro5hin
o enternone
o Beorn
o Also by antizeus


Display: Sort:
Abuse of Comment Ratings | 41 comments (41 topical, editorial, 0 hidden)
thresholds (3.50 / 6) (#1)
by jesterzog on Wed Dec 13, 2000 at 03:05:11 AM EST

I quite like the threshold idea. It's probably correct that some idiot's going to create multiple accounts but until that starts happening, I don't know if it's worth worrying about.

Another thing I think might be quite good is to maybe ignore the highest and the lowest rating of any given comment. Maybe just the lowest if people think it's only the troll voters we want to block.


jesterzog Fight the light


don't think so... (3.00 / 5) (#2)
by pope nihil on Wed Dec 13, 2000 at 03:12:46 AM EST

Ignoring the single lowest rated comment has an even worse potential to be abused by people with multiple accounts.

Personally, I barely ever look at the comment ratings here. It's not like there are tons of mindless miscreants taking over (knock on wood), so most of the comments I read seem to have some value.


I voted.

[ Parent ]
Outlier exclusion (2.00 / 1) (#32)
by kmself on Wed Dec 13, 2000 at 06:57:06 PM EST

Sorry, I'm not sure I understand what you mean by your comment that "Ignoring the single lowest rated comment has an even worse potential to be abused by people with multiple accounts". How, exactly?

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

well (4.00 / 1) (#40)
by pope nihil on Wed Dec 13, 2000 at 10:40:35 PM EST

they only have to have 2 accounts to achieve the desired effect (rating an unrated comment to 1).


I voted.

[ Parent ]
Democracy (3.36 / 11) (#3)
by enterfornone on Wed Dec 13, 2000 at 03:18:17 AM EST

I think I bought up most of this in my Death of K5 story a while back.

The problem as I see it is Rusty feels the need to have a completely democratic system. I think democracy is working for the story system, but certainly not for the comment system.

The solution, rather that calculate trust based on comment ratings, scrap the ratings and have Rusty appoint people who he knows he can trust who have the power to kill spam etc on sight.

The other solution which I think I've bought up before is to use a slashdot style descriptive mod system (ie troll, informative etc.) rather than numbers. With such a system it is much easier to spot when it is being abused.

--
efn 26/m/syd
Will sponsor new accounts for porn.
I disagree (2.50 / 2) (#24)
by pb on Wed Dec 13, 2000 at 03:48:28 PM EST

Actually, I think the comment ratings here would scale better. The problem very well might be that we don't have enough people on K5. Secondarily, I'd still like to add accountability into the moderation system, if possible; I wanted this on slashdot too.

Also, I'd like to point out that slashdot is not an example of a successful moderation system. I'd *love* to be able to moderate there, and do my part, because their current system of choosing moderators sucks. I've been on slashdot for a long time, and believe me, it's only gotten worse.
---
"See what the drooling, ravening, flesh-eating hordes^W^W^W^WKuro5hin.org readers have to say."
-- pwhysall
[ Parent ]
slashdot (2.50 / 2) (#26)
by enterfornone on Wed Dec 13, 2000 at 05:18:07 PM EST

Yes, their system of choosing moderators sucks because they don't have enough moderators to control the volume of spam. But (spaking as someone who metamods daily) it's rare that posts are modded down unfairly.

--
efn 26/m/syd
Will sponsor new accounts for porn.
[ Parent ]
Slashdot problems... (2.50 / 2) (#36)
by pb on Wed Dec 13, 2000 at 08:10:50 PM EST

They will never have enough moderators to control the volume of spam. In fact, a lot of the spam exists to take away mod points from the people who moderate down spam. And if it's posted at 0, then you probably shouldn't waste a point on it. (mod up the good posts, Luke...)

What I notice is that the wrong posts get moderated up. Post early, post something funny, post something interesting but barely relevant... and you might get moderated up. Post something on topic that discusses the story, and you might very well get ignored. In that respect, it's just about the opposite of kuro5hin.
---
"See what the drooling, ravening, flesh-eating hordes^W^W^W^WKuro5hin.org readers have to say."
-- pwhysall
[ Parent ]
true (2.50 / 2) (#37)
by enterfornone on Wed Dec 13, 2000 at 08:15:43 PM EST

But that is mainly due to volume, not aflaw in their mod system. It doesn't change the fact that the descriptive system works better than a numeric system for the comments that are moderated.

I think if everyone could moderate every comment then Slashdot's system would work.

--
efn 26/m/syd
Will sponsor new accounts for porn.
[ Parent ]
Descriptive or Numeric (2.50 / 2) (#38)
by pb on Wed Dec 13, 2000 at 08:38:01 PM EST

Well, boiling everything down to one number is a simplification, but at least that number converges to a value.

Also, allowing fractions helps too. On slashdot, people will argue over whether something deserves a 5, or if it's just a 4 or a 3. Also, they have non-descriptive moderations like "Overrated" and "Underrated" that aren't subject to Metamod.

And making the mods descriptive doesn't solve everything either. A post can easily be both "Funny" and "Offtopic", or "Insightful" and "Flamebait". But the mod system forces it to go one way or another. And slashdot has its own little quirks that allow you to get a (+5, Flamebait).

I still like the old idea of sorting by controversy; that is, ranking posts on slashdot by the number of mods, and not necessarily the sum of the positive or negative mods. Or maybe some compromise in between.
---
"See what the drooling, ravening, flesh-eating hordes^W^W^W^WKuro5hin.org readers have to say."
-- pwhysall
[ Parent ]
Re: Democracy (3.00 / 1) (#30)
by kagaku_ninja on Wed Dec 13, 2000 at 06:37:30 PM EST

I posted my solution to this issue during one of the previous rehashings of this topic. Naturally, it was never rated...

Anyway, the essence is that to be democratic, everyone should be allowed to moderate. In order to prevent the tyranny of the majority (and to prevent attacks from hostile parties), each user should have the ability to choose which moderators opinions are used to produce his or her view of the message database.

[ Parent ]
Great idea.. (3.00 / 1) (#33)
by enterfornone on Wed Dec 13, 2000 at 07:01:06 PM EST

but how would it work in practice? Do you simply say "I trust Rusty to rate comments fairly, but I don't trust Inoshiro" or do you have a system that automatically finds like minded people (based on the people you rate up and down).

--
efn 26/m/syd
Will sponsor new accounts for porn.
[ Parent ]
Re: Great idea.. (2.50 / 2) (#35)
by kagaku_ninja on Wed Dec 13, 2000 at 07:37:50 PM EST

The system is more complex, and requires more computation to generate each page. I don't know how kuro5hin is currently implemented, but this is a way to do it:

In the database, attach a list of moderator_id/rating pairs to each message, as opposed to an single 0-5 aggregate rating.

In my view preferences, I select the moderators I want, as well as other preferences (kill lists, weights, how much weight to give to moderators not on my list, etc)

When I go to view a topic, the ratings for each message get computed dynamically, based on my moderator preferences.

This also requires the ability to view the ratings assigned by moderators, in order to rate moderators.

[ Parent ]
Trust metrics (4.08 / 12) (#4)
by raph on Wed Dec 13, 2000 at 03:27:04 AM EST

I'm going to blow my own horn here. Making comment ratings based on a trust metric similar to Advogato's would largely solve abuse problems.

Such a trust metric could be set up as essentially an access control system, as it is on Advogato, or could be set up on a finer grain, so that input to the moderation system is itself the flow that runs through the capacity constrained network.

Group trust metrics work. There is a sound mathematical proof of their attack resistant properties, now backed up with over a year's experience on Advogato. With all the moderation systems for Web-based Usenet clones out there, somebody should try the Reese's Pieces experiment of using them to make moderation more trustworthy. I'm not sure if K5 is the best testing grounds for this. It is, unfortunately, not trivial to program.

In the somewhat hostile environment of today's Internet, nobody should be surprised that any metadata scheme without the attack resistance property fails to be trustworthy.

Trust is good, but you can only trust it so far (3.60 / 5) (#12)
by kmself on Wed Dec 13, 2000 at 06:55:34 AM EST

And trust me, Raph, Advogato was a major influence on both me and Rusty in putting Scoop together. Thank you for the inspirations.

I'd like to see a trust metric play a role in what Scoop can do, and believe that it will help address some issues. I believe it would be helpful to start creating some trust-based capabilities within the system. Despite a popular belief here, I don't see Scoop and/or K5 as necessarily being an all-voices-are-equal forum. But the discriminatory system I'm shooting for is a meritocracy. Equality is highly overrated.

In fairness as well, Advogato has been around for over a year, but at a population and activity level far smaller than Kuro5hin or more pointedly Slashdot. Size matters. I believe many of the more critical issues in a site occur as it scales, and particularly as people start assuming entitlements to site features. While Advogato has a highly evolved trust metric, its forum, presentation, and content scoring capabilities are not as advanced as Scoop's. Whether this has positively or negatively effected site content, I'm not so sure.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

I don't see a problem. (3.66 / 9) (#5)
by Inoshiro on Wed Dec 13, 2000 at 03:42:23 AM EST

If everyone gives a fair rating based on the content quality, vs. personal bias, this will not be a problem. So what if one person "targets" someone? Everyone is free to say the given rating should be different and add their voice.



--
[ イノシロ ]
The problem lies ... (4.22 / 9) (#9)
by StrontiumDog on Wed Dec 13, 2000 at 05:28:09 AM EST

... not so much in the first level comments, where reader density is high, but in the subthreads, where reader density decreases with each conversation level. Your post, for instance is rated; mine is likely to remain unrated. Usually after three or more branches, comments cease to be rated, and anybody going through the "Comments By" for a particular user is effectively determining the rating for that thread.

The percieved problems arise when someone is targetted and given 1.00 scores on every posting accessible from the "Comments By" section, regardless of content. This can occur out of vindictiveness, or simply trolling (I have seen users get 1.00 scores even though their postings were well-written and appropriate, and it happens even to posters who are generally non-confrontational in nature).

This may mean more modding from k5 readers but the volume of articles, comments and diary entries is such that few k5 readers, if any, can cover 100% of all articles.

It is true that the only consequence of losing Trusted User status is the loss of ability to see and moderate trash. However scoring articles also serves a social purpose in k5: it is the equivalent of "Hear, hear" or "Boo", and prevents the need for "Me too!" or "You suck!" postings in order to make your opinion known. This also means that flamebaiters stand much less chance of generating lots of heated but empty responses: people make their verdict known by giving a low score to a flamebaiter and moving on. It is this social function that is being distorted by the 1.00 modders.

[ Parent ]

Regression to mean and Mojo problems (4.87 / 8) (#11)
by kmself on Wed Dec 13, 2000 at 06:47:53 AM EST

Good analysis of the problem. It's leafnode and editorial comments which appear to be the most significant points of abuse. This is where several factors come in to play:

  • Limited readership -- not everyone reads editorial comments and/or leafnode comments. Editorial comments, in particular, have a short TTL, and are frequently not subject to further scruitany after an article is posted. This is bad.
  • Personal vesting. Both leafnode and editorial comments are subject to moderation by those who've taken a stand on an issue. While this doesn't mean abuse always exists, it suggests a stronger swing of opinion in moderating.
  • Regression toward the mean. Simple probabilities suggest that more frequently moderated comments will tend to regress toward a mean value. Outlier values are more likely where fewer moderations (or, particularly, one moderation) exist.
  • Mojo computation. Mojo is currently computed based on the mean of comment moderation, plus a decay factor, which I'm going to ignore for the moment. Eg: for five comments, moderated at 1, 1, 2, 3.33, and 4, mojo is computed as 2.27. If the actual moderations (which we can treat as n times the comment's mean moderation) are: 1x1, 1x1, 1x2, 10x3.33, and 4x4, then a mojo based on actual moderations granted to the user rather than the moderation of comments, is now computed as 3.02.

The decay factor acts in such a way that more recent comments count more than less recent ones. I believe Mojo is currently based on activity within the past 60 days or 60 comments (these are configurable parameters). Moderation weight is decremented by one for each comment's moderation calculated -- it decays from $max_comments to $max_comments - number-of-comments. Or if you prefer code:


sub calculate_mojo {
    my $S   = shift;
    my $uid = shift;

    my $max_days   = $S->{UI}->{VARS}->{mojo_max_days};
    my $max_comments = $S->{UI}->{VARS}->{mojo_max_comments};
   
    my ($rv, $sth) = $S->db_select({
          WHAT => 'points, cid, sid',
          FROM => 'comments',
          WHERE => qq|uid = $uid AND ((TO_DAYS(NOW()) - TO_DAYS(date)) <=
$max_days) and points IS NOT NULL|,
          ORDER_BY => 'date desc',
          LIMIT => qq|$max_comments|,
          DEBUG => 0});
   
    my ($sum, $count);
    my $weight = $max_comments;
    my $real_count = 0;
    while (my ($rating, $cid, $sid) = $sth->fetchrow()) {
          $real_count++;
          $count += $weight;
          $sum += ($rating * $weight);
          $weight--;
          warn "\tFrom cid $cid, Story $sid, rating is $rating: \n\tCount:
$count, weight: $weight, Sum: $sum\n" if $DEBUG;
    }
    $sth->finish();
   
    my $new_mojo = ($sum / $count) unless ($count == 0);
   
    warn "New mojo for user $uid is $new_mojo\n" if $DEBUG;
    return($new_mojo, $real_count);

What's needed is for the line

$sum += ($rating * $weight);
to read
$sum += ($rating * $weight * $mods);
...where $mods would be the number of moderations on particular comment.

By the way, a problem you don't mention is the mutual admiration society over in trolltalk where everyone's moderating everyone else to '5'. Well, almost everyone. This is essentially the web-of-mojo attack I'd suggested Mojo might be susceptible to -- if the TT kids decide they want to play it out. So far they actually seem to enjoy things here.

There are also some other ideas to ponder.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Compulsory comments (3.77 / 9) (#6)
by evvk on Wed Dec 13, 2000 at 03:44:08 AM EST

I've also noticed that people may tend to give low ratings just because they don't agree with you or whatever reason while others may give good ratings. I'd like them to argue and justify their rating; too often you get bad ratings but no one tells why. Therefore I think the system should be modified so that others than trusted users should have to leave a comment/reply for low ratings (say, 1 and 2)? If a user just leaves a blank comment, we know he's an idiot; if he has nothing to say, the rating cannot be justified.

Submission queue (3.83 / 6) (#13)
by kmself on Wed Dec 13, 2000 at 07:06:34 AM EST

The changes I'd like to see to the submission queue are:

  • An editorial queue, at least for some topics (Op-Ed, Columns, F&P, Culture). A lighter submission-only queue for News, MLP, and lighter topics would also be appropriate.
  • No topical comments in submission queue. This is where material should be honed for submission. This goal has been completely lost. People will post topical comments anyway. However, these comments will be closed to discussion once the story posts. Fixing some bugs in the voting process (below) will also alleviate the protracted waits we've seen for some stories.
  • Mandated comments to rate a story in an editorial, and possibly a submission, queue. The editorial and submission queues are modeled after an editor's bull room -- you have to say why you like or dislike a piece, you can't just pan or praise it.
  • An expedited voting system with post, dump, and quorum thresholds. First of the post or dump thresholds to be crossed would determine the outcome of the story. If the quorum is reached without a decision, the story is dumped. This should avoid the interminable waits we've seen in queue.
  • Author kill option. That's the author gets to kill the story, not voters get to kill the author. Thought this does give me an idea....
  • A "post-by" deadline. Some material is timely and isn't worth posting after this deadline expires.

The idea of having trust-based roles is another think I've been kicking around. I'm not sure that just anyone should be able to vote on stories.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

problem with that (3.60 / 5) (#17)
by mikpos on Wed Dec 13, 2000 at 09:58:09 AM EST

I think the system should be modified so that others than trusted users should have to leave a comment/reply for low ratings (say, 1 and 2)?

First of all, I should say that a comment with a rating of 1 is not a bad comment. For example, if someone were to reply to this particular story of with a comment like "I don't think this is a very big problem", I would probably rate it as 1 (or maaaybe 2, depending on my mood and/or the phase of the moon). It's not that it's a really bad comment, such that I'd feel annoyed at the poster for wasting my time; it's just that it's not as good as some of the others. I guess I'm just saying that bad is relative, and what's bad on k5 is usually pretty good.

That said, my bigger problem with your suggestion is that if I vote something down to 1 or 2, it's because, by definition, it doesn't incite discussion. It seems like an awful waste to have to write "your comment doesn't add much insight; I doubt if any serious discussion will start from it" over and over again for each comment I mod as 1 or 2.

[ Parent ]

Accountability (3.77 / 9) (#7)
by GreenEagle on Wed Dec 13, 2000 at 04:16:12 AM EST

I'm not sure about the overhead of this solution, but what about simple accountability ?
There would be a moderation totals page for each comment, where you see who moderated with what score.

The current system is completely anonymous, that's why it is being abused IMHO.

A few suggestions (4.00 / 10) (#8)
by Beorn on Wed Dec 13, 2000 at 05:04:13 AM EST

We need to look at the purpose of the rating system before it can be improved. As I see it, ratings has two functions: sorting by relevance (noise reduction) and reader feedback. I read Kuro5hin many times a day, with comments sorted by time, so the first function is irrelevant to me.

I'm more interested in feedback from the people who read my comments. I always missed that on BBS's and Usenet, because there are many reasons why people who likes or dislikes what you write would never bother to reply. Voting is a very easy and effective form of reader feedback, sort of like the boos and cheers of the British Parliament.

To sort comments, you need a simple, average value to sort by. (This can be improved easily with vote count tresholds.) But for author feedback, averages are too inaccurate. I would like the ability to see the number of votes for each rating, (as a link from the average rating.) This would be simple and provide useful information. It would also reduce the power of grudgeful Anonymous Cowards.

One last thing, though: It's not worth it to retune and reinvent and reimplement the comment rating system forever. If the rating system becomes too complex or too fragile to be useful, it must be discarded. No ratings at all are better than ratings that doesn't work.

- Beorn

[ Threepwood '01 ]

There needs to be some kind of accountability (3.88 / 9) (#10)
by Simon Kinahan on Wed Dec 13, 2000 at 06:11:47 AM EST

I have noticed a distinct tendency for comments to be rated according to how much people agree with them. There have been a few occasions, in political threads, where I've noticed people, including myself, being rated down for having opinions which don't fit in with someone's "party line". I have not problem being disagreed with, indeed thats pretty much the purpose of talking about politics, and indeed I'm even happier to discover if I'm actually wrong. But I want to be told - being wrong or disagreeable is not a reason for a comment to be hidden away at the bottom of the page with a 1.6 score.

Coupled with the fact that not everyone rates all comments, and readers tend not to read all the comments anyway, a single score of 1 because of a disagreement can mean a comment is read by almost noone.

Someone else suggested that comment ratings should be catalogued by user in the same way as story ratings. I think thats a good idea, but I also think ratings should carry some idea of why the comment was rated as it was. Maybe a little type in box next to the rating selector ?

Simon

If you disagree, post, don't moderate
Some bias is OK. Listing mods is an option. (3.25 / 4) (#14)
by kmself on Wed Dec 13, 2000 at 07:09:26 AM EST

Bias will enter into the moderation process, and there's no way around that.

Statistical tracking of who's doing what, to whom, when, etc., may help in developing broader metrics of trusts or identifying vendettas and mutual admiration societies.

Another option would be to have the option of revealing -- either to all, to trusted users, or to the comment submitter -- the UIDs of all who've moderated a particular comment. Accountability is good.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Re: Some bias is OK. Listing mods is an option. (4.25 / 4) (#18)
by YellowBook on Wed Dec 13, 2000 at 11:42:27 AM EST

On vendettas and mutual appreciation societies: I think the latter are much more dangerous. I'm assuming that at least some trusted users will read every comment attached to articles that they read, which is enough to keep vendettas from doing any harm. After all, one person's score can only have so much influence on the average score for a comment.

There's also a place for vendetta-like activities: keeping down trolls. If I see an egregious example of trolling (i.e., not just flame-baiting or posting deliberate misinformation, but something either abusive or grits-esque), I will not only score it 0, but will go to that user's comments page and search for other trolls from them to score down (though I will not score down legitimate posts from someone who also trolls).

Mutual appreciation societies are a bigger problem because they enable trolls to keep the average score of their posts high enough to be visible to real readers.

It would be interesting to have a "who rated this post what" stat available somehow.



[ Parent ]
More on topic at Scoop (2.00 / 1) (#27)
by kmself on Wed Dec 13, 2000 at 06:09:38 PM EST

I've written more on this subject in a Scoop forum . I believe this addresses some of your concerns.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

I disagree. (4.08 / 12) (#15)
by Mr. Piccolo on Wed Dec 13, 2000 at 07:56:18 AM EST

Unlike that other site, everybody has moderator access all the time. Therefore, if everyone were to be active in the moderation system all the time, these "rogue" moderations would get lost in the underflow. Of course, that isn't possible.

I believe the only "fix" that the system needs is for more people to moderate, and in addition people need to feel free to over-moderate if they believe a comment has an unfairly high or low score. If you see a comment rated at 2 that you feel deserves far more than that, don't hesitate to give it a 5! Conversely, if you see a grammar flame at 5, give it a 1 if you think it's unfair!

Ultimately, the success of the system depends on the percentage of "coward moderators" to justice-seeking moderators.

The BBC would like to apologise for the following comment.


See Raph's post (3.66 / 3) (#28)
by kmself on Wed Dec 13, 2000 at 06:22:41 PM EST

Scoop certainly is open to attack, and large sites such as K5 are probably more susceptible than most, particularly with open subscriber policies. Simply: it's possible for rogue users to create multiple accounts (granted that multiple emails are required) and influence the moderating scheme. The incentives issue tends to run against the white hats as well. This forum itself appears to have been subjected to some moderation abuse -- several posts definitely deserving of high-4 ratings are floating nearer 3. Attacks are likely to come in waves and not be disruptive all the time -- the larger user and moderation base probably does help. However, as I said in my response to Raph, trust and trust-based privileges are likely to be necessary at larger sites.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Yeah. (3.42 / 7) (#16)
by pb on Wed Dec 13, 2000 at 09:45:21 AM EST

I saw this happen briefly as well, and it is annoying. Mostly I think it's the lack of accountability that disturbs me.

But I can't think of a good solution offhand; they would all involve more overhead, or fascist guidelines.

The other issue is, of course, why can we moderate comments that are outside of stories in the first place? As Merekat suggests in my Diary entry, why can anyone else rate the comments in *my* Diary?

Originally I thought there weren't going to be ratings at all in the Diaries, but I could understand if only the author moderated them, or more importantly, if only the author could choose who moderated them. (maybe three options: no one, me, or everyone. That's until we implement Unix-style groups. ;)
---
"See what the drooling, ravening, flesh-eating hordes^W^W^W^WKuro5hin.org readers have to say."
-- pwhysall

Anonymity _is_ the root of these problems (3.33 / 3) (#23)
by Zukov on Wed Dec 13, 2000 at 03:03:56 PM EST

There are always a few people who get their kicks from destroying/vandelizing the work of others. In the "real" world, that behaviour often results in getting smacked.

On K5, there is no penalty, no risk of discovery. Like Nelson says, "It's a victimless crime, like punching someone in the dark"

I suppose all solutions involve (vast?) additional overhead, but being able to

  • See moderator identity
  • Being able to add/remove the input of a particular moderator from your own individual scoring system (trust)
  • Having stats on which moderators agree with each other consistently, to weed out people having multiple accounts
  • Being able to add/remove posters from your own individual comment view

    would make it possible for each reader to pull the weeds from their garden..


    ȶ H (^

    Yes, I have just bumbled upon Gnome Character Map. Please ! me.
    [ Parent ]

  • Bark-bite mismatch (3.33 / 3) (#34)
    by kmself on Wed Dec 13, 2000 at 07:18:42 PM EST

    If it's any consolation, annoying at this is, the impact after the most recent posts is minimal. Mojo focusses on recent activity, so that such a mass downrating of a user has relatively thin effect.

    Still, you're right, it's a bug, and there should be means to identify such activity. It also funks up comment prioritization in older forums, which makes life more difficult for those who do find time to read them.

    OTOH, I'll occasionally roll through older material and find myself rating as I go.

    --
    Karsten M. Self
    SCO -- backgrounder on Caldera/SCO vs IBM
    Support the EFF!!
    There is no K5 cabal.
    [ Parent ]

    Why I don't moderate (3.40 / 5) (#19)
    by Yer Mom on Wed Dec 13, 2000 at 11:48:06 AM EST

    When I have Rate turned on, this POS machine crashes even more than usual. I suspect the strain of having to draw all those pulldowns causes Windoze to run short on resources. It's only a problem on longer discussions, but it is a problem. Without the rating boxes, I can get away with it until the discussion gets really long.

    I guess having Display set to Nested doesn't help, but still... I'd like an option to break a discussion up into pages, say by setting a maximum number of comments per page. That would rock.

    Hm, maybe I should post this as a diary entry so I could attach a poll to see whether it's my fault for using Windoze or not :)
    --
    Smoke crack. Worship Satan. Admin Unix.

    Oooh! Oooh! (3.33 / 6) (#20)
    by Elendale on Wed Dec 13, 2000 at 12:37:19 PM EST

    This may have been said before, but i think its a pretty good idea. See, when you post a comment that gets (for example) one five moderation it may be a 'five' but it isn't (IMHO) as 'strong' a comment as a 4.33 that has been rated three times. That's currently, anyway. What about making Mojo not count the final rating of the comment, but instead each of the individual ratings (or some form of (comment rating) * (number of ratings) = mojo value). This seems to me to be an excellent solution, what are the problems that i overlooked? Can anyone see a better solution? It seems to me this would make both an individual rater marking everything as '1' (had this happen to me a few times too, but only a few comments) would be less effective and it would be incentive to rate '5' comments with another five (stopping abuse later)- assuming the deserve it. What i mean is, when a comment is rated '5' people aren't likely to rate it if its not going to be a lower score- not necessarily a bad thing- but when someone comes along later and rates your five a one, even the best comment in the world turns into a three rating.

    -Elendale
    ---

    When free speech is outlawed, only criminals will complain.


    Y'know (3.00 / 3) (#25)
    by Elendale on Wed Dec 13, 2000 at 05:15:21 PM EST

    I just thought of something, though. When i see a comment at '5' and don't think it belongs at a '5' and end up rating it a 4 or even a 3 to bring it down (or conversely, see a '1' and rate it a '5' to take it to a 3 or so) the system i described above would count this as another rating. I suppose one way of fixing this is to rate for a good reason. That is, when you rate you would have to say "I think this comment deserves a score of 'x'" and hope others rated it too. Still, its a problem i didn't see at first and i'm not so sure its a good idea anymore.

    -Elendale
    ---

    When free speech is outlawed, only criminals will complain.


    [ Parent ]
    Add'l stats badly needed (3.00 / 3) (#31)
    by kmself on Wed Dec 13, 2000 at 06:39:09 PM EST

    Yes, additional stats are badly needed, and are discussed elsewhere in this discussion, on K5, and at Scoop by myself, Rusty, and others. Check comments posted by us for more info.

    Your followup regarding "adjustment" moderation is also on point IMO. I'll give a 5 to a low-rated comment even if it's only deserving a moderate value if I feel it's been unduly moderated down.

    --
    Karsten M. Self
    SCO -- backgrounder on Caldera/SCO vs IBM
    Support the EFF!!
    There is no K5 cabal.
    [ Parent ]

    Glitch? (4.00 / 1) (#41)
    by Kunstwerk on Thu Dec 14, 2000 at 01:22:01 PM EST

    That's a good idea, but I think there's a glitch with it. Correct me if I'm wrong.

    The glitch is: Under a rating system like yours, mojo whores would make sure to post early and often to make sure their comments are rated by a lot of people - a lot like what occurs on Slashdot, where any idea posted by a semi-sentient humanoid will get a +3 if it's among the first 10 comments.

    What I like about K5 is that a good comment which comes in late in the discussion will have just as much value a a late-comer... and I think a system like yours would take some of that away.

    --KW [Diary] /* Do all humans pass the Turing Test? */
    [ Parent ]

    Descriptivity and/or accountability (3.87 / 8) (#21)
    by Erf on Wed Dec 13, 2000 at 12:44:53 PM EST

    As mentioned by others already, the biggest problem with the ratings system is that it's just a number. More description would be nice for the rating to mean something. Showing the number of times the comment was given a 4 or a 1 or whatever would be helpful. /.-style descriptive ratings (insightful, redundant, well-put, me-too, etc) would, I think, give the rating much more meaning, as it forces you to say why you're giving the rating. (Maybe two separate list-boxes: the rating #, and the reason...?) Right now people rate either the "quality" of the comment, or the extent to which they agree with the comment ("me too!"). Perhaps showing all the reasons people rated the comment would really show what people think about it. (You know, having "agree" and "disagree" buttons might be cool... maybe not, but it's a thought...)

    Even just showing the number of times the comment was given a 4 or a 1 or whatever would be interesting -- I'd trust that 2 or 4 I get a lot more if 17 people gave it that...

    The accountabilty idea is also interesting, but I'm more ambivalent about that -- it opens the rater up to personal attacks by an offended ratee (or by other raters). I don't know if this would be a problem here, but it's something to think about, and watch for, if rating accountability is brought in.

    ps. Yes, "descriptivity" is a word. It is now, anyway. :)

    -Erf.
    ...doin' the things a particle can...

    My opinion is... (3.16 / 6) (#22)
    by Fireblade on Wed Dec 13, 2000 at 01:10:57 PM EST

    if you build an idiot proof system, the world will just produce a better idiot.

    I don't think it is possible to design a comment rating system that could not be easily abused by anyone with the time and desire to do so. The question is, does this problem warrant the time and aggravation required to design a system that is, at best, less easy to abuse?

    Here's another question. Would I be any better than a coward rater if I were to pick one of the victims and rate all of their comments at 4 or 5?

    Maybe it's just better to ignore them and hope that they grow tired of this and move along.

    Yes (3.00 / 3) (#29)
    by enterfornone on Wed Dec 13, 2000 at 06:28:40 PM EST

    That is my main problem with the comment rating system. It encourages people to vote as to whether or not they agree with a comment, so the lazy can voice their opinion with a click rather than taking the time to reply. This does not encourage discussion.

    --
    efn 26/m/syd
    Will sponsor new accounts for porn.
    [ Parent ]
    Two things (3.75 / 4) (#39)
    by trhurler on Wed Dec 13, 2000 at 09:57:17 PM EST

    First, a technical solution, such as a threshold, will not work. Creating and using multiple accounts is not difficult.

    Second, you don't need one. You need participation. Lots of raters means that one ass doesn't matter.

    By the way, this happens to a lot of people. If you say unpopular things, it will happen to you. If you in any way annoy certain people, it will happen to you. That's life. The bright side is, it almost never lasts, and usually that sort of moron isn't a trusted user, so at least you can't get 0 bombed easily. However, participation is not just an aid in solving this problem - it IS the solution.

    --
    'God dammit, your posts make me hard.' --LilDebbie

    Abuse of Comment Ratings | 41 comments (41 topical, 0 editorial, 0 hidden)
    Display: Sort:

    kuro5hin.org

    [XML]
    All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
    See our legalese page for copyright policies. Please also read our Privacy Policy.
    Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
    Need some help? Email help@kuro5hin.org.
    My heart's the long stairs.

    Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!