Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

[P]
sid=moderation

By rusty in News
Sat Feb 26, 2000 at 02:40:00 PM EST
Tags: Kuro5hin.org (all tags)
Kuro5hin.org

If you're logged in, you might notice that the "Not Yet Working" notice is gone from the comment rating controls. This is because, as far as I know, it is working now. I thought I'd post an explanation of how comment ratings work, and the theory behind them, both to explain what's going on, and to begin a discussion on rating and moderation in general, and this system in particular.


The basic operation of the new rating system goes like this:
  • Every logged in user may choose to participate in the comment rating system, or not. To participate, simply select "Yes" from the selector next to "Rate?" on the comment control bar, at the top or bottom of any comments page.
  • If you select yes, then you may rate every comment (except your own) with a score from 1 through 10 (lowest to highest). Simply select the score, and click "Rate" below the comment you'd like to rate.
  • Rating a comment '0' does nothing. It does not register a score of zero from you, it simply returns without changing anything. It's the default NULL value.
  • The total rating of a comment is simply the average of all ratings it's received.
  • After you've rated a comment, the selector below that comment will be set to the last rating you gave it. To change your rating, just select a different score and click "Rate" again.
  • You only get one vote toward the average, but you may change it as often as you like. Your new vote simply overwrites the old one, and recalculates that comment's overall rating with the new value.
The goal of the rating system is not to punish the bad, or really even to reward the good, but merely to provide a way of ordering the comments based on a community "quality" evaluation. This is not a thinly-veiled troll punishment system.

Why can you change your rating? I felt that ratings shouldn't be static. They need to reflect a discussion, which is an evolving and changing beast. A comment which is thoughtful and informative early on in the discussion may be overshadowed later by other comments which address the same issue, but provide more information. I wanted to allow people to move comments around, not just "spend points."

There are a couple of problems that I anticipate with this system. First, someone suggested adding the ability to tag comments as you read through them, and bulk moderate them all with one-click (Oops! Patented phrase! I meant "with 0 < clicks > 2"). This is a good idea, and I would have done it already but I was 98% finished with the current system when I read that, and it'll require a little retooling. But expect that to appear in the next few days.

The other problem I see so far is that there's no way to indicate why you rated a comment the way you did, and then to filter based on reasoning. Is a comment a 6 because it's funny, or informative? I expect this will be a useful addition to the system-- the ability to register a description of the comment as well as a numerical rating, and then you can choose the order you'd like comments to appear in based on their "average description." This idea is still not entirely clear in my head, so I'd really like feedback on it.

Do we need moderation here yet? No, not really. Without exception, you have all been stellar contributors so far. We could get along just fine without it. But I felt that it would be a nice thing to have, if it does it's job right, and that eventually, we will need it, so it's best to experiment with it now, when things are still under control. :-)

If moderation and ratings are of interest to you, please use this as an open forum for ideas and thoughts on "quality ordering systems" in general. It'll be on my hotlist, so let me know what you think!

Sponsors

Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure

Login

Related Links
o Also by rusty


Display: Sort:
sid=moderation | 68 comments (68 topical, editorial, 0 hidden)
Re: moderation was done well (2.25 / 4) (#1)
by ramses0 on Sat Feb 26, 2000 at 03:00:12 PM EST

I really like your idea of each user posting an explanation with their moderation: example: [rated 8, funny], and when filtering eventually happens, you get the waay cool option (hopefully) of saying...

set my defaults to show all +5 comments, except if they are funny they must be +8. It'd be tricky to get the UI on the preferences pages done right, but it'd be neat to play with, imho.

--Robert
[ rate all comments , for great justice | sell.com ]
Re: moderation was done well (3.00 / 1) (#5)
by rusty on Sat Feb 26, 2000 at 03:18:11 PM EST

Yeah, that's something like what I have in mind. Right now you can order separately by date and by rating, so if I set "Newest First" and "Highest Rated First", the comments will be ordered by rating, and then by date when the ratings are equal. What I see is another modifier, where I can say: Order by rating, then, when ratings are equal, show "informative" comments, then "funny" comments, etc., then when the other two factors are equal, order by date. There would be a list of categories on your prefs page, and you could arrange them most desirable to least, I guess.

But I like your idea, too. You could have separate thresholds for each "flag." The question is how to determine what a comment's flag is. Do you just take the most commonly applied flag to be the right one? You can't really average a value judgement like that. Any ideas?

____
Not the real rusty
[ Parent ]

Re: moderation was done well (none / 0) (#8)
by ramses0 on Sat Feb 26, 2000 at 03:37:20 PM EST

Well, this is a little gross, but whenever anybody rates a comment, they give it a numeric, and a category [+8, funny].

If someone rates it [+8, funny] and someone else rates it [+4, insightful], then... well, the average overall rating would be 6, but the average funny rating would be 8, and the average insightful rating would be 4.

So that's what you'd have to work with, however, I think it would be complicated or performance-degrading to actually implement. I'm sure that if you worked at it for a couple of days, you wouldn't have any problems.

--Robert
[ rate all comments , for great justice | sell.com ]
[ Parent ]
Re: moderation was done well (none / 0) (#10)
by rusty on Sat Feb 26, 2000 at 03:49:54 PM EST

If someone rates it [+8, funny] and someone else rates it [+4, insightful], then... well, the average overall rating would be 6, but the average funny rating would be 8, and the average insightful rating would be 4.

What about this-- you could choose the comment's flag based on a weighted comparison of ratings and flags. So, in your example, the comment would have flags of: 4x(insightful) and 8x(funny). So that would determine that the comment was, overall, "funny".

If some more people rated it, say someone came along and said "6, insightful" and someone else said "1, funny". Then the flags would get averaged, so the comment's per-flag averages would be:
((4 + 6) / 2) x "insightful" = 5, "insightful" and
((8 + 1) / 2)) x "funny" = 4.5, "funny"
Which would change the comment's overall flag to "insightful". Did that make sense?

I don't know if it would be better to determine an overall flag, or just set a per-flag average and let people sort based on each flag.

____
Not the real rusty
[ Parent ]

Re: moderation was done well (none / 0) (#12)
by ramses0 on Sat Feb 26, 2000 at 03:56:06 PM EST

It made perfect sense, it just becomes a matter of how much disk-grinding you're willing to do, and how much processor time you're willing to devote.

I'd advise to wait until at least tomorrow before you start hacking, you never know what good ideas are going to come up about moderation. :^)=

--Robert
[ rate all comments , for great justice | sell.com ]
[ Parent ]
Re: moderation was done well (4.00 / 1) (#13)
by Nyarlathotep on Sat Feb 26, 2000 at 03:57:01 PM EST

The seperate thresholds for each flag has the appeal of seeming ``more accurate,'' but I would like to remind you that these are all ad-hoc algorithms so ``more accurate'' could actually make the systems mistakes worse, i.e. some people never rate anythg better then 5 so they pull down the average.

I would recmmand having 3 flags (funny, interesting analysis, informative) which are rates seperatly, but have only a yes or no status andn ot the full 1-10 rating. this would mean all post would recieve a ratio (goods / over total rates). This solves the problem of funny/interesting/informative being orthogonal directions AND the that 1-10 decissions are going to be much more subjective then keyword based decissions.

I do not htink there is any need to make comment raters post a comment, but their ratings should not be anonymous, i.e. maybe a buttong you press to see the ratings of a comment.

Campus Crusade for Cthulhu -- it found me!
[ Parent ]
Re: moderation was done well (3.00 / 1) (#14)
by rusty on Sat Feb 26, 2000 at 04:26:09 PM EST

I'm confused about the first two paragraphs of your post. What is the ratio composed of?

I do not htink there is any need to make comment raters post a comment, but their ratings should not be anonymous, i.e. maybe a buttong you press to see the ratings of a comment.

I agree with this, actually. I was thinking last night that maybe there should be another link "view ratings" that appears with comments. But where should it display? Would a pop-up window be annoying? I think I'd be inclined to just make a small popup that displays a list of users and the ratings they currently have registered for that comment. If the window was still open, and you clicked a different comment's "show ratings", it'd replace the content in the popup (so you don't get an infinite number of little windows.

Is there a better way of doing this? I normally dislike popup windows, but I can't think of a much better way to display that.

____
Not the real rusty
[ Parent ]

Re: moderation was done well (3.00 / 1) (#32)
by Nyarlathotep on Sat Feb 26, 2000 at 06:29:44 PM EST

I'm confused about the first two paragraphs of your post. What is the ratio composed of?

I was saying that the 1-10 rating may compicate things. I've never read any psychological research on it but I do not think people do a very good job of 1-10 type ratings. Instead you could just ask them "Was it funny?" and the average rating wuld be the percentage of people who said yes. It's just probable better idea to ask people to answer a simpler yes/no question.

I don't think the drop down vs. pop-up makes much diffrence. The pop up looks better to many people but requires JavaScript (i think) which some people only turn on when they need it. It's really sounds like an even trade off to me.

I would say it is just as importent to have a way to get the ratings of a specific person so that users can produce evidence that a specific person is being a bastard. The one thing that might make the drop down really cool would be having it link to the coresponding users previous ratings. This would make it really easy to find out about people.

BTW> I noticed you talking about changing it so people could rate all the posts at once. If you do not want to overhaul your forms layout (i.e. you want to have seperate form tags for each message) you could offer a JavaScript option for ratins, i.e. pressing the ratings buttong opens a pop up which says "thank you" and dose notrelode the whole page. This is sorta a cheeper solution for mass ratings, but it should probable be optional since some people don't like JavaScript (just face it.. JavaScript is dangerous if you are going to visit those porn sites<grin>).

Campus Crusade for Cthulhu -- it found me!
[ Parent ]
Re: moderation was done well (none / 0) (#36)
by rusty on Sat Feb 26, 2000 at 07:36:40 PM EST

I see what you're saying. The problem with yes/no is, what questions do you ask? A comment can be different levels of quality, for different reasons. I think a rating + flag system would work-- that would combine the "reasoning" with the "ranking" portions of the judgement.

I'm leaning tward the drop-down list, personally. I don't like using javascript on public sites, especially for things that everyone should have access to. Roll-over lightups are one thing-- if they don't work, it doesn't matter (unless you're idiotic and make roll-overs a crucial part of your site navigation, which don't even get me started on that). Maybe you could choose which you like better in your prefs.

I like the idea about linking to a page of a user's ratings. The idea that, like story moderation, I didn't want comment moderation to be anonymous has been irking me, but I couldn't think of how to make it non-anonymous. Showing per-comment mod lists, and linking to per-user lists is an excellent way to do it, I think.

About bulk-moderation-- javascript again. Don't really want to go down that road. I don't think rewriting the forms will actually be a problem, really. There will still be a submit button next to each comment, and one at the top and bottom. Hitting any button will make all your choices up to that point go into effect. It'll actually make the code a little smoother, I think. The problem with not reloading the page is that you don't get the benefit of your moderations till you do reload it, so I imagine everyone would probably want it reloaded anyway...

____
Not the real rusty
[ Parent ]

Re: moderation was done well (4.00 / 1) (#46)
by Nyarlathotep on Sun Feb 27, 2000 at 12:57:00 PM EST

The problem with yes/no is, what questions do you ask?

The 4 I recommended would be "Was it funny?", "Was it more relevant information?", "Was it a good analysis?", and "Was it ontopic?" I don't know what else you would really need to ask. Actually, you could combine interesting and informative. These three make a nice attempt at providing an orthogalal basis on which to judge a comment. Note: People would mark their thresholds for each of these seperatly. I guess the most importent thing is to take whatever system you do take as provisional, i.e. see how it works and try and figure out what could be improved after we gain some expeernece with it.

Campus Crusade for Cthulhu -- it found me!
[ Parent ]
Another possibility... (4.00 / 1) (#23)
by rusty on Sat Feb 26, 2000 at 05:31:52 PM EST

What if all ratings were put in a drop-down list below each comment. This wouldn't be part of the form, just a convenient way of displaying a list of names and ratings for this comment. There would also a preference selector for whether you want to display this info or not.

____
Not the real rusty
[ Parent ]
Drop-down ratings -- good idea (2.00 / 1) (#55)
by kmself on Sun Feb 27, 2000 at 06:26:47 PM EST

I think I like the idea. Implementation might need some work. Problem with a drop-down box is that you're transferring the moderation information with each post, though it's likely to be accessed only infrequently. Some other means of showing moderation history -- a link? -- might be better.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Re: moderation was done well (none / 0) (#30)
by jawad on Sat Feb 26, 2000 at 06:20:13 PM EST

Keep in mind -- The reason (well: a damn good reason, but not *the* reason) Slashdot doesn't have people moderate everyone's posts is that it'd be too damn time consuming. I'm new to Kuro5hin (and imagine my surprise finding sid=moderation to be the newest article), and I'm not that familiar with the amount of replies to expect for an article, but I would imagine, it'd be a little bit of a bitch to handle.

Just thoughts from a newbie, though.

[ Parent ]

Re: moderation was done well (none / 0) (#31)
by rusty on Sat Feb 26, 2000 at 06:27:39 PM EST

You don't *have* to moderate, remember. Some people are interested in moderating, some aren't. In fact, if you don't want to moderate at all, you can choose "No" on the comment control bar, and you won't even load the moderation controls. If you want to have the option, choose "Yes", and moderate only when you feel a comment is particularly good. You can also set your sorting to ignore comment ratings altogether, if you don't like moderation at all.

Is there some other way I can make people feel like rating is an option they can choose to engage in, rather than a burden they have to deal with?

____
Not the real rusty
[ Parent ]

Re: moderation was done well (3.00 / 1) (#35)
by ramses0 on Sat Feb 26, 2000 at 07:27:36 PM EST

First, a semantic issue: I think the term "rate" is more appropriate for what ~we~ are doing to comments. We are assigning scores based from 1-10, thusly, assigning a rating.

I think "moderate" is a good term to use for stories. Merriam Webster (http://www.m-w.com/) lists one of the definitions o "moderate, verb" to be: "to preside over or act as chairman of", and that's exactly what we are doing with the story queue.

Onwards and upwards... the "rating system" is a little-bit "in-your-face" right now. It interrupts the "look and feel" of kuro5hin comment viewing, which isn't that great. Maybe you could have a "yes|no|sometimes" option, and when a person hit "sometimes", it would put a link [parent|reply|rate] after each comment.

This would allow users to rate comments when they wanted, but also keep it out of the way, and still allow complete and blissful ignorance. ;^)=

--Robert

[ rate all comments , for great justice | sell.com ]
[ Parent ]
"Rate" vs. "Moderate" (3.00 / 1) (#37)
by rusty on Sat Feb 26, 2000 at 07:42:55 PM EST

Yes, "rate" is completely the word for it. I try to remember to call it that, but sometimes I slip into old (bad) habits.

About putting the "rate" link in there. Wouldn't that be kind of a pain, to have to reload the page every time you wanted to rate a comment? I want it to become just second nature, you read a comment, you set the selector, just out of habit. When you're done reading, you click the submit and let your opinions be known. I know that having form elements everywhere isn't great, but that's kind of the price you pay for implementing a syetm like this. Is there another way to provide the same functionality of the current system, i.e. always-available, instant rating, without the burden of the forms?

____
Not the real rusty
[ Parent ]

Re: "Rate" vs. "Moderate" (none / 0) (#42)
by ramses0 on Sun Feb 27, 2000 at 12:56:01 AM EST

I think we're on the same wavelength... just need better communication :^)=

1) Always available rating (as you have it now) is good. I like. It's convenient, and has much utility.

2) For "casual" moderators, I think the "separate page", only rate one comment at a time method would be good. It doesn't get ugly, and [parent|reply|rate] are good controls to have at the bottom of each comment.

3) I agree with whoever said: "comments should start with 'rating: none'", and user preferences should determine how those comments are displayed.

...waxing philosophical for just a moment (please hold tight ;^)=

If only one person rates a comment, and they happen to rate that comment at the right(*) level, then the world is truly a harmonious place. One person has done work, and the entire community has benifitted by it.

(*) right meaning: the consensual value desired by the community.

Since the basis of your website/software is "trust your users", moderation could actually be a pretty rare thing. Assuming that each member of the community thinks with the same harmonious mind as the rest of the community, then only one person would ever rate any one comment (good for users, and good for the database).

It is only when a comments has been ~erroneously~ rated (as judged by the community at large) that more than one person should have to rate it.

The more I think about your software, the more elegant it is turning out to be ;^)= Great work.

(but please, a favor!
http://www.kuro5hin.org/?op=displaystory;sid=2000/2/14/16194/3934#79)

--Robert

[ rate all comments , for great justice | sell.com ]
[ Parent ]
Re: sid=moderation (3.00 / 1) (#2)
by Paul Dunne on Sat Feb 26, 2000 at 03:03:52 PM EST

Is the rating number supposed to be floating point? If so, why?
http://dunne.home.dhs.org/
Re: sid=moderation (4.00 / 1) (#3)
by rusty on Sat Feb 26, 2000 at 03:11:26 PM EST

Yes, it's supposed to floating point. Mainly because in a 1 - 10 system, many averages won't be whole numbers, and I though rounding to the nearest int might be a little clunky. Basically just for a little more precision. They'll always be rounded to two decimal places, so you don't have to worry about seeing (Rating: 3.1415926535 8979). Do you think they should only be integers? Does it matter?

____
Not the real rusty
[ Parent ]
Re: sid=moderation (none / 0) (#4)
by Paul Dunne on Sat Feb 26, 2000 at 03:17:48 PM EST

If I was programming it, I would probably have made it an int, and taken a laissez faire attitude to rounding (I forget how perl does this, but C just truncates down, I think?) -- but then I always was a very lazy programmer...
http://dunne.home.dhs.org/
[ Parent ]
Re: sid=moderation (none / 0) (#6)
by rusty on Sat Feb 26, 2000 at 03:23:15 PM EST

Yeah, the problem really is that perl's approach to rounding is.. umm, bad. The only thing you can do without math libraries (AFAIK) is truncate down to an int (like C-- I think it probably just passes through to the underlying C math handlers). This wasn't very satisfying to me, so I added a 'use Math::BigFloat;' and calculated the average with the fdiv function, which allows you to set the number of decimal places to round to. This was, actually, easier than just being lazy, and setting the 2 significant digits was exactly as easy as not setting them, so I just went ahead and did it.

Remember, the three Virtues of a programmer are Laziness, Impatience, and Hubris. :-)

____
Not the real rusty
[ Parent ]

Re: sid=moderation (3.00 / 1) (#7)
by Paul Dunne on Sat Feb 26, 2000 at 03:35:07 PM EST

> Remember, the three Virtues of a programmer are Laziness, Impatience, > and Hubris. :-) Hmm, yes, check, check, and check: got all three in abundance. It is thus a mystery to me why I wasn't a better programmer! I know a bad workman blames his tools, but I put it down to having spent most of my programming career using the, ah, inimitable RPG/400. Even SQLWindows was a great relief after that. And to think the book that decided me on becoming a programmer was "The Soul of a New Machine"! (Yes, I *know* that's mainly about hardware engineers).
http://dunne.home.dhs.org/
[ Parent ]
Re: sid=moderation (none / 0) (#67)
by dvwood on Thu Mar 02, 2000 at 07:33:38 PM EST

I prefer integers (for simplicity), but given a poor rounding mechanism, than just one decimal point. That is all that will be useful.

D
Just visitin'
[ Parent ]
Re: sid=moderation (2.00 / 1) (#9)
by rongen on Sat Feb 26, 2000 at 03:45:06 PM EST

If I was doing the moderation system....

Which I'm not! :)

Seriously, I am designing a time-tracking app which uses statistics to
measure earned-value, and accuracy of project estimates and stuff
(designing, not building). Consequently I have been thinking a little
about issues like relevance and ratings. Here is what I think about
the requirements I have seen above:

1. There should be a default NULL (or whatever) value for both rating
and reason. This way the person only moderates by choosing to do
so. If you moderate one comment and then submit your moderation it
only affects the one comment. The user would not have to rate each
comment, nor choose not to moderate because they didn't want the
hassle of reading all comments (although I consider a complete
reading to be really important before posting/moderating, if
possible).

2. For the "reason" moderation. Come up with a gradation that starts
at "totally off topic (TOT)" through "hilarious" and goes to
"extremely relevant" (or whatever). Each one could be represented
internally by a numeric value (um, I guess you could have assumed
that). Moderation on "reason" could then be averaged like the
numeric ratings. Since the gradation is a relevance and funniness
"spectrum" this would be pretty easy to do (If 5 people think it's
funny and one thinks its TOT then it's probably actually funny).

Just my two cents!


read/write http://www.prosebush.com
Re: sid=moderation (none / 0) (#11)
by rusty on Sat Feb 26, 2000 at 03:53:50 PM EST

About NULL values: D'oh-- I forgot something important in the article. A rating of zero is "NULL". It doesn't do anything. It won't change your existing rating, and if you haven't rated yet, it won't register a rating from you.

Right now, rating is on a per-comment basis. When you click the "rate" button, you ONLY affect the comment directly above that button. Ultimately, with bulk ratings, clicking the "rate" button will change your rating on all comments that aren't set to 0. So, basically, it already works that way. :-)

____
Not the real rusty
[ Parent ]

Re: sid=moderation (none / 0) (#20)
by rusty on Sat Feb 26, 2000 at 05:11:22 PM EST

If I was doing the moderation system....

Which I'm not! :)

But you ARE! This is not that other discussion site. I want your opinions and ideas. If something doesn't work, if the readers don't like it, I will change it. What's the difference between a programmer and God? God doesn't think he's a programmer. :-) Seriously, I don't believe that I am God, or that I know all the right answers. I believe that we as a group (whatever group that is) can probably get closer to "the right" answers than any individual one of us. So, yes, you are in charge of the moderation system. Just like everyone else here. The only decree I will hand down from on high is that you all have a responsibility here to do what's right. Exercise your responsibility wisely. :-)

____
Not the real rusty
[ Parent ]

More Questions I Have (2.50 / 2) (#15)
by rusty on Sat Feb 26, 2000 at 04:32:20 PM EST

Does anyone have an opinion about the "initial score" commenys should get? Should it stay at one, like it is now? Should there be an "unrated" score (you will be able to order unrated comments first, by the way)? Should they start at 5, to land in the middle of the scale, and move up or down depending on what people think?

Hmm. I had more questions, but now I forget them.

____
Not the real rusty

Re: More Questions I Have (4.00 / 1) (#16)
by Demona on Sat Feb 26, 2000 at 04:39:12 PM EST

I'd prefer an initial non-integer, "Unrated" -- less assumption going into it. User preferences could determine how to display unrated posts.

[ Parent ]
Re: More Questions I Have (none / 0) (#19)
by kmself on Sat Feb 26, 2000 at 05:07:00 PM EST

A default "unmoderated" score would be useful for people who are looking for recent, but unmoderated, posts. A "zero" might be a useful internal representation, but the display value should reflect the unmoderated status.

The default moderation score should also be set to the same "unmoderated" value, as it appears that submitting a "moderate" vote moderates all currently visible posts. The current zero value, and vote resubmission option, essentially does this, so I don't think the functionality is different, but the user's interpretation is.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Re: More Questions I Have (4.00 / 1) (#21)
by rusty on Sat Feb 26, 2000 at 05:14:16 PM EST

Mentioned in the bullet list, in the article: Right now, submitting a rating ONLY affects the comment that rating button is attached to. With the advent of full-screen "bulk moderation", submitting a rating will rate all comments on screen. A zero vote is NULL, not 0. I should change the list to display a '---' or something instead of zero, so people don't get confused. But the lowest you can ever rate a comment is "one".

____
Not the real rusty
[ Parent ]
Re: More Questions I Have (3.50 / 2) (#22)
by rusty on Sat Feb 26, 2000 at 05:15:28 PM EST

I remembered another question. Do you think 1 - 10 is too many choices? Would something like 1 - 5 be better? This is easy to change, so if anyone has feelings one way or the other, let me know.

____
Not the real rusty
[ Parent ]
Re: More Questions I Have (5.00 / 1) (#34)
by Imperator on Sat Feb 26, 2000 at 07:24:50 PM EST

How about a range of 1-5 or even 1-4? That would make it easier for people to be consistent.

Also, how about some sort of graphical representation of the rating? Say, make the range 1-4, and then have 7 different images corresponding to increments of .5. That will make it much easier to see a post's rating without the overhead of reading/calculating/thinking. (Maybe display the numerical score, to one decimal place, beside it?)

[ Parent ]
Re: More Questions I Have (3.00 / 1) (#41)
by fvw on Sat Feb 26, 2000 at 11:54:10 PM EST

Without wanting to make the page look like a clownsuit: how about giving the bar above each comment a different color (or shade of blue) based on it's rating? That should make highly rated comments easily visible.

[ Parent ]
Re: More Questions I Have (4.00 / 1) (#38)
by FlinkDelDinky on Sat Feb 26, 2000 at 11:05:17 PM EST

I think 1 to 10 is to big of a range. 1 to 4 seems more manageble to me. Basically, a post is good or it isn't. A 1 would simply be a worthwhile read, 4 would be a rather rare and outstanding post.

Going from 1 to 10 is just to fine grained. I think most of us could 'find' a community consensus on just what a 1, 2, 3, or 4 post is. No way on 1 to 10.

BTW, most primitive languages have number for 1, 2, and 3. After that they just used a word that means 'many'. I'm saying this to support the intiutiveness of 1-4 with 4 being outstanding. It's just something we'll be able to parse better.

[ Parent ]
Re: sid=moderation (3.00 / 2) (#17)
by Anonymous Hero on Sat Feb 26, 2000 at 04:57:24 PM EST

If there's no karma to build up, what good is it?
(currently -25 at Slashdot)

Re: sid=moderation (4.00 / 2) (#18)
by rusty on Sat Feb 26, 2000 at 05:05:19 PM EST

Karma is silly. I've said it before, I'll say it again. Think about it-- what has it done for slashdot? Two things:
  • It's given some people incentive to post according to what will get moderated up (karma whoring), which is clearly not always the same as what is good.
  • It's given other people incentive to post the worst trolls and spammage they can think of, to see how low they can get their karma to go (i.e. (currently -25 at Slashdot)).
It's only use there, really, is to regulate the pool of potential moderators, to a coarse degree. There's no need for that here, since you're all moderators, any time you want to be. So karma, IMO, would be a big Loss for everyone. I don't think it does anything positive at all.

That's the longer version of "karma is silly". :-)

____
Not the real rusty
[ Parent ]

Re: sid=moderation (4.00 / 1) (#29)
by jawad on Sat Feb 26, 2000 at 06:16:56 PM EST

It's also made people more careful in saying what they say. Very few people are interested in having really low karma's, most are interested in maximizing their karma's.

A lot of people say a lot less of the crap they'd be inclined to say before moderation was cumulative (i.e. karma.)

yes, i'm that jawad.

[ Parent ]

Re: sid=moderation (3.00 / 1) (#33)
by rusty on Sat Feb 26, 2000 at 06:36:18 PM EST

A lot of people say a lot less of the crap they'd be inclined to say before moderation was cumulative (i.e. karma.)

I don't think that's true. First, observation: If karma makes people say less crap, why is there so much more crap being said there than ever before? Second, through reasoning: Karma is not public. No one knows what your karma is. The most they can ever deduce is that if you post at +2 by default, it's over 30. A secret, non-public valuation of me is never going to affect my behavior. The only way, I think, people would really care about karma is if it were open for all to see (like, posted in the subject line of every one of your comments, not just available on the user page). When it's secret, it just becomes a game to see how much or how little you can accumulate.

That's my take on it anyway. I think it's, in theory, a good idea, but one that didn't pan out for them, whether through bad implementation or some fundamental flaw in the reasoning.

____
Not the real rusty
[ Parent ]

Re: sid=moderation (3.00 / 1) (#40)
by FlinkDelDinky on Sat Feb 26, 2000 at 11:36:21 PM EST

You know, I never post on /. but I read it all the time and never knew what the karma thing was about.

Now that I know I just got an idea like karma. Maybe when a user makes a post that post will be pre-rated the average of that posters prior posts.

If the post doesn't live up to the prior posts it can be rated down.

But here's the kicker, the rating the post receives will replace the average.

Example: my average rating is 1.8. I post to an article. Automaticcally it's rated at 1.8.

You log on and only want to see articles > 1.5. You don't think my post deserves a 1.8 so you rate it a 1.

No that post is rated a 1, thereby ensuring the rest of the > 1.5 crowd aren't annoyed by my lousey 1 post.

BTW, I'm using my 1 to 4 scale. I posted about 1 to 4 vs. 1 to 10 in a prior reply to you.

BTW, in this 1 to 10 is 1 average or is 5 average?

I've just thought of a neccessary addition to 1 to 4 and that is -1 for a bad post.

[ Parent ]
Silly karma? (none / 0) (#51)
by kmself on Sun Feb 27, 2000 at 06:21:41 PM EST

I'm not quite sure I agree.

Slashdot's implementation of karma has a number of problems, including that it is based on a voting (rather than ranking) scheme, and is only very slowly responsive to recent activity. This allows someone to build up karma over time, then "spend it down", slowly, in a long trollfest.

Using a ranking vs. moderating scheme as is being done at Kuro5hin offers a couple of alternatives for implementation, including both how karma is accumulated, and how it is implemented.

Suggestion: "karma" is equal to some weighted average of post rankings. Weighted to favor recent activity (there are several smoothing algorithms, including geometric smoothing, which would be appropriate). Decay should be based both on post volume (less recent posts weigh less) and a time basis (extending back say a year?).

The effect would be to set the initial post value of a registered use, with weight=1. Essentially, your karma becomes your own rating score for your own post, but it doesn't weight itself (for the current post) any more than any one else's rating, so that a high-karmic troll could be quickly reduced to below average (3) by two "1" rankings.

This would give a person an incentive to post meaningful stuff, cap the maximum score (karma can only asymptotically approach 5), and tends to be more immediately reflective of recent activity. It also allows a person who posts infrequently but meaningfully to have a high karma, and would tend to limit the karma of someone who posts frequently but only gets roughly average rankings.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Re: Silly karma? (3.00 / 1) (#54)
by ramses0 on Sun Feb 27, 2000 at 06:26:02 PM EST

What would be the benefit of having karma, though? why is karma a good thing to have for a discussion site?

I think your ideas are great, and well-planned, well-thought-out, but what would scoop/kuro5hin gain by a metric called "karma". You can go to a user's personal site (ie: http://www.kuro5hin.org/?op=user&tool=info&uid=275), see their most recent comments, along with their most recent scores, would this solve the problem that you're thinking of?

--Robert
[ rate all comments , for great justice | sell.com ]
[ Parent ]
Re: Silly karma? (4.00 / 1) (#56)
by rusty on Sun Feb 27, 2000 at 06:33:52 PM EST

Well, in this case, the benefit of "karma" would be to provide an initial ordering for posts, from which user ratings would proceed.

This is, I agree, a much better way of handling karma, and I think it might actually even work. I still have two problems with it though:

  1. Value v. Work: I think this would be a bit of work to implement, and I don't see the value being worth the effort
  2. The Special Case: I don't want to always have my posts start at a high rank, sometimes I just have some goofy little joke to make. So, for this case, I'd have to implement a self-chosen "don't use my default score" as well. See #1. :-)
Basically, I want things to be exactly as complex as they need to be, and no more. You can choose to see unrated comments first, if you want, so we shouldn't have the "good post buried in noise" issue. If you say something interesting, it'll get rated up.

I'd need more convincing as to the immediate benefits of karma, still, before I'll consider doing it.

____
Not the real rusty
[ Parent ]

Iron the bugs out first (none / 0) (#59)
by kmself on Sun Feb 27, 2000 at 07:58:02 PM EST

What I think we've got is a platform from which to build out the idea of karma, and something resembling agreement on how to do it, but no pressing need at the moment. A default/inital post placement would be a nice thing to have (the karma bonus), but isn't necessary at the moment given post volumes. File the idea in the can and work on getting other stuff out first. When it is needed, we'll know how to proceed with it. What I'd be happier in seeing in the meantime are filtering tools to go with the post-ordering and rating stuff.

WRT default post level -- this could be addressed by allowing you to moderate your own post -- to a maximum level specified by your karma. What this buys you? I don't know. Maybe in this case down-ranking yourself doesn't count against karma, but a cumulative low-rating score by others does. But that sounds complicated.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Re: Silly karma? (3.00 / 1) (#65)
by dvwood on Thu Mar 02, 2000 at 07:08:42 PM EST

The question is, what should good karma get you? Automatically higher ratings on your posts? No. Definitely not. Everybody occasionally has a lame/empty post. What Karma should give you, if you really want to have an award system, is the capability to either disable/enable moderation. If 5 is the best Karma someone can get, and 1 is the lowest, then perhaps anyone with Karma at 2 or less does not get to moderate.

Also, in arriving at a person's Karma setting, perhaps it should be an average of the last 10 post rankings. If someone doesn't have 10 posts, then they default to 3 (neutral).

It's simple. It's relatively responsive to recent activity. It offers a rational reward.

Dvw


Just visitin'
[ Parent ]
norming, ranges, scales, stats (none / 0) (#24)
by kmself on Sat Feb 26, 2000 at 05:34:02 PM EST

Issues regarding the ranking range:

  • +/- 10 points is a bit broad. Since the display/rank value is floating point, with a sufficiently large moderation base, the law of large numbers will make individual score granularity irrelevant.
  • It's not clear how a post should be scored. The way I read it, 5 is median value, anything less than 5 is sub-par, anything above is favorably rated. Looks like most people here are sticking to score=1.
  • I'd suggest a zero-centered score ranging +/- 2, corresponding to strong disapproval, mild disapproval, neutral, mild approval, strong approval.
  • Having a number of scales might also make sense, though this gets cumbersome. I think three might be appropriate, say information, humor, and civility:
    Info: -- - 0 + ++
    Humor: -- - 0 + ++
    Civil: -- - 0 + ++
    ...I think that covers most of the relevant ground. It would also allow me to filter out posts which are grossly uncivil, filter in posts which are tremendously funny, and otherwise filter in reasonably informative posts. Which is how I'd generally like to read.
  • Adding a couple of additional statistical measures to the reporting might be intersting, and possibly help fine-tune filtering rules. Cumulative votes would indicate oveall interest. Standard deviation would indicate controversial issues (or system abuse with competing votin blocs).

Good start, IMO.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.

Re: norming, ranges, scales, stats (none / 0) (#25)
by ramses0 on Sat Feb 26, 2000 at 05:38:57 PM EST

I think rating a comment with a negative value is a bad idea, for the same reason that Anonymous Hero is better than Anonymous Coward

--Robert
[ rate all comments , for great justice | sell.com ]
[ Parent ]
Why? (3.00 / 1) (#26)
by kmself on Sat Feb 26, 2000 at 05:46:01 PM EST


--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Re: Why? (none / 0) (#28)
by rusty on Sat Feb 26, 2000 at 05:53:26 PM EST

That's what I was getting at in my post below. ramses0 put it more succinctly than I did. I chose "Anonymous Hero" rather than "Anonymous Coward" on purpose, because language affects us (I think). It might sound silly, but seeing yourself referred to as "Coward" repeatedly, even in a joking way, I think will affect your behavior eventually, or at least provide a bad clue as to how you are expected to behave. I think words are powerful, and should be treated carefully. This is approximately the same issue.

____
Not the real rusty
[ Parent ]
Re: Why? (none / 0) (#43)
by Paul Dunne on Sun Feb 27, 2000 at 04:08:08 AM EST

I must confess, I thought "Anonymous Hero" was a real user for a while.

Why have `anonymous' posting at all? If someone actually does want to be anonymous, because they work at MS or IBM or wherever, and don't want to be caught "spilling the beans", then they can create an account for the purpose: it's just as anonymous. I can't see any other real reason for wanting `anonymity' save this special case. Maybe it's just me, but I prefer to know who I'm talking to, even if all I can see is a `handle' and a maybe a web page in User Info.
http://dunne.home.dhs.org/
[ Parent ]

Re: norming, ranges, scales, stats (none / 0) (#27)
by rusty on Sat Feb 26, 2000 at 05:47:21 PM EST

About 1 - 10 scale, I think I agree. I'll probably change it to a smaller scale.

About 5 being median-- that's sort of how I see it. The default value doesn't match up with the rating system right now, I agree. Posts should start as "unrated" which is not "0", but just doesn't fall within the scale. After that, you're basically right, that the middle value of the scale will be average, and above and below will be better / worse than average, respectively.

I don't know about a -2 through 2 scale. Functionally, it's indentical to a 1 - 5 scale, but socially, the effect is a little different. Having minus scores seems indicates "disapproval", while having all scores be positive, I think, indicates "less good through more good." Yes, the distinction is totally semantic, but I think semantics are important. The idea is that, if a comment is here, it's worth being seen in some way (natalie portman and grits will just disppear). So the question for the rating system is to decide "most good" and "least good". Am I just splitting hairs or being too optimistic, or does anyone else think this makes sense?

____
Not the real rusty
[ Parent ]

kewl bug (none / 0) (#48)
by kmself on Sun Feb 27, 2000 at 06:03:59 PM EST

I guess this post's parent will be the highest-voted post ever on Kuro5hin ;-) Rusty reparented the rating scores but didn't adjust scores on existing comments. It's a minor bug as it won't affect articles posted dating forwards.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Re: kewl bug (4.00 / 1) (#52)
by rusty on Sun Feb 27, 2000 at 06:22:08 PM EST

Heh. :-) Not really a bug, just an issue I didn't feel like fixing. I mean, I'd have to adjust all the scores to fit the new scale... blah. Not worth the effort. I did make all rated 1 scores "unrated" in the assumption that they were just rated 1 because that was the default. That's as far as I'm gonna go :-)

____
Not the real rusty
[ Parent ]
Re: kewl bug (4.00 / 1) (#53)
by rusty on Sun Feb 27, 2000 at 06:26:02 PM EST

Just discovered another bug, while posting that comment. If you were in "minimal" mode, comment preview only showed you the title. Not very useful, I'd say. :-) Should be fixed now.

____
Not the real rusty
[ Parent ]
Another one... (none / 0) (#62)
by rusty on Sun Feb 27, 2000 at 10:37:38 PM EST

Forgot to change the user default points to '0' for everyone (so that all posts start out as "unrated" instead of '1.00'. Didn't anyone else notice this? :-)

____
Not the real rusty
[ Parent ]
Hey, we gotta leave *something* for you (none / 0) (#63)
by kmself on Mon Feb 28, 2000 at 12:15:21 AM EST

Yes, I'd noticed that. Figured it was in the works, though I was going to say something in a day or so if you hadn't got to it yet.

Also noticed that "none" sorts to the top, which is good, as it means unrated comments show up first. I was getting a bit peeved that my own posts were showing up with "1" at the bottom of the pile... <g>.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Re: Hey, we gotta leave *something* for you (none / 0) (#64)
by rusty on Mon Feb 28, 2000 at 12:56:49 AM EST

Also noticed that "none" sorts to the top, which is good, as it means unrated comments show up first.

Actually, that's just the default behavior. If you investigate the other comment sorting select boxes, one of them lets you sort be "Unrated, the Highest" "Highest rated first" (which will sort unrated to the bottom), "Lowest rated first" (which will also sort unrated to the top, and then up from that), and "Ignore ratings." I'll post an update to the story, probably tomorrow, explaining the new stuff.

____
Not the real rusty
[ Parent ]

moderation with confidence (4.50 / 0) (#39)
by emjay on Sat Feb 26, 2000 at 11:28:26 PM EST

Given any thought to a confidence interval along with the rating? The more people vote, the more narrow the interval becomes. That way you could set it up in your user preferences to only show comments rated at (for example) +5 with a +-1 degree confidence. Just something I think would be cool.
-------------------------
We can't stop here, this is bat country!
Re: moderation with confidence (4.00 / 1) (#58)
by ramses0 on Sun Feb 27, 2000 at 06:38:49 PM EST

This is a nice idea... (I'm gonna respond to the comment immediately following it as well, cuz i'm lazy ;^)=

I like this idea a lot, but I like even more the the idea of "controversial" comments.

Controversial comments are often the ones most worth reading, and Phillipp's idea is a nice, easy, simple way to determine where controversy lies.

I like it better than "moderation with confidence" because if a post is definitely a 1 or a 10...er...5, then it will never become "confident", and it is a less useful measure for readers than "controversy".

"confidence" would probably be a really good statistic for archival purposes. Both of these are really good ideas.

--Robert
[ rate all comments , for great justice | sell.com ]
[ Parent ]
Controversial comments (none / 0) (#44)
by Philipp on Sun Feb 27, 2000 at 05:33:33 AM EST

What I see as the biggest abuse of rating is that people tend to downrate comments that are legitimate and reasonable just because they do not agree with its stated opinion. If you ever followed the fate of a comment critical to the party line at Slashdot, you know what I mean: it gets up and down like a yoyo. No matter how much you urge moderators to rate unbiased from opinion, they will do that. Check the review comments in the story submission queue for examples.

Here now my idea: You can measure how controversial a comment quite easily by computing the variance of the scores. If all people rate a comment 5, it has a variance of 0. If one half rate it 9, the other half rate it 1, the variance is 4 (or 4 square? I never know this...). Hence, a controversial comment has a higher variance in ratings. This should be displayed, as numeric value and/or with the addition of "controversial" (variance 1-2), "very controversial" (variance >2). I am making these numbers up, but you get the idea.

There should be then an option to choose to see only comments above X, plus controversial ones (variance bigger than Y).

PS: I would be prefer a 1-5 rating than a 1-10. First of all there should be a proper middle, and then I would have a hard time to figure out if a specific comment is 7, or if it really good enough to be regarded 8. Bad, weak, ok, good, great. That are enough choices.

alias kn 'killall -9 netscape-communicator'

The thing about moderation is... (4.00 / 1) (#45)
by Strange Charmed One on Sun Feb 27, 2000 at 08:52:50 AM EST

...it works iff people moderate- about 1/5 of the comments have been moderated on this topic about moderation. (And I am responsible for about half those moderations)

I would also say that starting the default at 1 is a bad idea- either make it 5 or unrated- it gives people the impression that mostpeople haven't liked the unrated comments- again the psycological nuances you (Rusty) appear to be so fond of.

I would also say that calling people "Anonymous Heroes" is saying that we are NOT /. too strongly- and looks as if we are trying too hard to not be /., and brings up the /. name in my mind. My preference would be something neutral like "visitor".
--
Feel the urge to put excessively cute little quotes into your .sig?

JUST SAY NO!

If you or one of your friends is frequently plagued by this tendency, Help IS available- Ask me how.

Participation will grow with volume (none / 0) (#47)
by kmself on Sun Feb 27, 2000 at 05:58:01 PM EST

As the number of posts increases, and the relative value of moderating increases, I suspect you'll see more moderation happening. We've only had a couple of days to shake this out yet, and the overall post count is low.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Re: Participation will grow with volume (2.00 / 1) (#66)
by dvwood on Thu Mar 02, 2000 at 07:28:38 PM EST

I could see fewer posts getting more moderation, as people exhaust the posts and decide to rate, while having a great number of posts encourages people to spend their time browsing and not rate.

And anything that really strikes a chord will be frequently rated.


Just visitin'
[ Parent ]
Bug: ranking range check (none / 0) (#49)
by kmself on Sun Feb 27, 2000 at 06:10:30 PM EST

Rusty reparented the ranking range from 1-10 to 1-5. This happened between cache refreshes on my browser, so I had one view of the article with ranking ranges at each level. I can apparently post ranks from either scale, just did this with the "If there's no karma" post, currently ranked "10".

The ranking scheme may be subject to spoofing if a range check isn't provided to limit scores to the stated range.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.

Re: Bug: ranking range check (3.00 / 1) (#50)
by rusty on Sun Feb 27, 2000 at 06:12:52 PM EST

D'oh. Good call. I made this same silly mistake with story moderation the first time that went up too. You'd think I never learn. :-)

____
Not the real rusty
[ Parent ]
Re: Bug: ranking range check -- fixed (none / 0) (#57)
by rusty on Sun Feb 27, 2000 at 06:34:43 PM EST

Should be fixed now.

____
Not the real rusty
[ Parent ]
Minimal rocks! (1.50 / 2) (#60)
by kmself on Sun Feb 27, 2000 at 08:02:44 PM EST

Just discovered "minimal" mode (see the "sort" drop-down) from one of Rusty's followups. That rocks. This was the format used by the old InfoWorld Electric forums, makes following long and convoluted discussions much more tolerable (though it usually involves multi-window browsing).

Bitchin' site, Rusty, rock on!

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.

Re: Minimal rocks! (5.00 / 1) (#61)
by rusty on Sun Feb 27, 2000 at 10:33:13 PM EST

I did it all for you, man. :-)

Seriously, though, minimal mode was indeed inspired by IWE. I kinda like it myself. I intend on letting you set the display mode based on the # of comments (so, 0-20 comments == "nested", 21-50 = "threaded", 50+ = "minimal") as well as "comments per page" thresholds for each mode. Be prepared, user prefs are going to get more interesting soon. (It's about damn time, too!)

____
Not the real rusty
[ Parent ]

Funny, civil, informative, controversial (5.00 / 1) (#68)
by dvwood on Thu Mar 02, 2000 at 08:10:16 PM EST

Rusty,
Great progress on the site!

There is value in letting people sort by these qualities, but it's important to keep it simple, and I'm not sure what a good solution is.

I think it's important to have one score for each post. One number. And I really like being able to designate at what number you care to view. There should be other ways to designate funny, controversial, etc ...

For me, informative is the more useful. Perhaps one way to measure this is to have the drop down rating measure only for informativeness (the most important quality), but to also provide checkboxes to designate posts as either funny or uncivil. People could specify a sort order for posts based on the numerical informative rating (say, show me all posts with a score above 2). And in some global user preferences section they should be able to set it so that they can additionally see all posts rated funny (even with lower scores than their informativeness threshold, say 2) and so that they can not see posts that are considered uncivil (based on a minimum number of people indicating a post is not civil, say 5 or 10 different people for example).

So when they go to the "View additional user preferences" link:
This page gives them capability to include all funny posts regardless of score (if a minimum number of people rank a post as funny).
This page gives them capability to turn off uncivil posts (if a minimum number of people rank a post as uncivil)

D
Just visitin'
sid=moderation | 68 comments (68 topical, 0 editorial, 0 hidden)
Display: Sort:

kuro5hin.org

[XML]
All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!