Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

[P]
The Trouble With Ratings

By Eloquence in Meta
Wed Aug 08, 2001 at 07:58:32 AM EST
Tags: Kuro5hin.org (all tags)
Kuro5hin.org

"Why did you rate this comment a 1? Surely it didn't deserve a 1! After all, it was three paragraphs long."

The FAQ states (as of this day) that comment shouldn't be rated based on whether the reader agrees with their content, but rather on their "presentation". If we treat the FAQ as a sacred, unchangeable document of binding rules, then the aforementioned fictitious complaint would be completely justified. We read complaints similar to it every day on K5. They pollute the comment space, they are "rating noise". The authors are wrong and should be rated down. The FAQ is grossly incorrect and illogical in its description of ratings as well. Here's why.


Why do we need ratings anyway?

Ratings tend to lead to different kinds of noise in discussions. Complaints about bad ratings, reasons for these ratings, stories about the rating system etc. This kind of meta-noise is almost always annoying for the majority of participants in the site. Add to this the emotional frustration of receiving bad ratings and you have to answer the question why ratings should be used at all.

There are several reasons for the existence of ratings. The aforementioned "emotional frustration" is partially intended since it discourages spammers, trolls and flamers. With the zero rating effectively making a post invisible to its author, the point of posting insults, bad links or other undesired content, getting a wide reaction from the site's audience, is entirely lost -- even if there is a reaction, the poster will not be able to observe it without first gaining trusted user status with an additional account. So ratings combined with reputations are a very efficient protection against abuse.

Furthermore, ratings make it possible to sort posts according to their quality (we get to the definition of this term next). In a lot of stories, many comments will simply consist of verbalized emotions, half-knowledge or actual disinformation. Surely a general rating system will not make it possible to truthfully, objectively sort all comments, but it should give the reader certain reading priorities.

What is quality?

There is no absolute truth in human knowledge, but there are approximations to the truth. Those who see the world as entirely the result of purely subjective experience, a construct of our minds alone, fail to explain why some ideas seem historically to work better than others in improving the conditions of our lives. The method with which you access and read this article is part of this innovation through understanding. Objective quality -- which, again, is not and does not claim to be absolute -- exists, and through peer review we can try to find it.

Rating comments can be seen as essentially a voting process in which the readers decide how well the comments describe reality. If they have no prior opinion or knowledge on the subject at hand, they should abstain from rating comments since their rating would only reflect the post's presentation and not its content. It is hoped that the majority will be able to decide properly on which comments best describe reality, since on any given issue, it is assumed that the majority has the proper education to make this decision.

This is, obviously, a wrong assumption, at least in our current society. So a rating system which is based on majority rule will often have undesirable outcomes. People with no knowledge on a subject will still want to have their "votes counted", especially if they are convinced by a comment's presentation. A better rating system would be much more complex. It would include not only the rating of comments, but also the rating of persons. It would allow peer groups with specific interests to form and to select (and group) experts into teams which, together, select content of good quality (and persons to join them in their particular field of expertise).

But this is not the rating system we are using on K5, so we should make the best of what we have. A simple fix to some problems associated with the current system may be to change the rating in such a way that it aggregates votes into a total value, so that comments get a total score (with no upwards limit) and the best comments are "selected", rather than elected -- this would avoid the emotional frustration associated with bad ratings while maintaining the aspect of quality selection. It would not be as personalized as a trust-based rating system, but better than what we are using.

If we want to keep using what we are using, however, the voting process outlined above seems to be quite reasonable, as compared to the alternative, which is:

The Myth of Emotional Disagreement

The FAQ suggests exactly the opposite to the rating method described above. As of today, it says this (it was changed in the past and will likely be changed again):

"Try not to vote based on whether you AGREE with the comment or not, or whether you like the person posting the comment. Try to base your ratings on HOW the comment is presented. We understand that this is hard, and that most of us are human and let emotions come into play."

Let's say there is a story on the front page that discusses the Holocaust and restitutions related to slave labor. In this story, we find a comment that has the following content:

America's leading gas chamber expert, Boston engineer Fred A. Leuchter, carefully examined the supposed "gas chambers" in Poland and concluded that the Auschwitz gassing story is absurd and technically impossible.

Leuchter is the foremost specialist on the design and installation of gas chambers used in the United States to execute convicted criminals. For example, he designed a gas chamber facility for the Missouri state penitentiary.

In February 1988 he carried out a detailed onsite examination of the "gas chambers" at Auschwitz, Birkenau and Majdanek in Poland, which are either still standing or only partially in ruins. In sworn testimony to a Toronto court and in a technical report, Leuchter described every aspect of his investigation.

He concluded by emphatically declaring that the alleged gassing facilities could not possibly have been used to kill people. Among other things, he pointed out that the so-called "gas chambers" were not properly sealed or vented to kill human beings without also killing German camp personnel. (note 20)

Dr. William B. Lindsey, a research chemist employed for 33 years by the Dupont Corporation, likewise testified in a 1985 court case that the Auschwitz gassing story is technically impossible. Based on a careful on-site examination of the "gas chambers" at Auschwitz, Birkenau and Majdanek, and on his years of experience, he declared: "I have come to the conclusion that no one was willfully or purposefully killed with Zyklon B [hydrocyanic acid gas] in this manner. I consider it absolutely impossible."

Sources:

The Leuchter Report: An Engineering Report on the Alleged Execution Gas Chambers at Auschwitz, Birkenau and Majdanek (Toronto: 1988). Available for $17.00, postpaid, from the IHR.

The Globe and Mail (Toronto), Feb. 12, 1985, p. M3

Now, according to the guidelines in the FAQ, this is a real gem of a comment, "pure gold" (let us ignore the fact that it was copied from Holocaust revisionist Ernst Zündel's website). Anyone who would rate such a comment down must be acting on their primitive emotions. It is eloquently written, comprehensive, includes sources and even buying information! It is a good counterpoint well-presented. What more could we ask for?

How about truth?

In this case, almost everyone knows that the comment is erroneous and would be able to easily present a refutation. By rating the comment up, however, those without a clue could easily be fooled by the rating into believing its content, especially if there is no reply to it. We can be quite sure that in this case, the comment would be quickly voted down, perhaps even zeroed (although some people, out of principle, would certainly fetch it up again by giving it a 5 if it crossed the threshold of invisibility).

Presentation should obviously not be any criterion for rating, exactly the opposite is true! The FAQ itself suggests rating up comments which make a "good point" -- what is a good point if not a post that is logical and true to our knowledge? The ridiculousness of the suggestion to rate based on presentation rather than content could not even be consistently kept up by its author through a few paragraphs.

A comment which presents some little-known truth, even if not well-written, should be rated to the top, while revisionist bullshit like the one quoted above should quickly fall to the bottom where it belongs. This is especially true since eloquence, as an attribute of a comment, will soon be determinable even by machines. Generally, a lengthy comment would almost always have to be voted up because of the effort put into it, so it might as well be given a higher starting value determined by automatically counting its words in such a scenario.

Reality is different anyway.

Most people have long realized that voting up comments according to their presentation makes no sense whatsoever. That's why most people rate comments up based on whether they agree with them. But because of the consequences (complaints, perhaps even mails to the editors of the site), few people have the guts to vote comments down which are nonsense.

The result is that the rating system is used not as a mechanism to vote on comments, because of the resulting emotional frustration, but rather as a mechanism to select the comments which are considered good. But as was pointed out before, the rating system isn't built for this kind of selection. Scores are averaged on a scale from 1-5. Now, if one group of users selects one half of the comments and rates them as 4 or 5, and another groups of users selects another half of the comments and rates them as 4 or 5 (the part they agree with), the result is that every comment is somewhere in the 4-5 range, perhaps with the shorter comments in the 3-4 range. Incidentally, this is the situation on K5 right now (with individuals instead of groups, of course), which makes the rating system mostly worthless, were it not for the few users who dare to use the ratings on the lower half of the scale.

The proper rating discipline

If you want to rate in a way that improves the discussion quality on the site, here's my advice:

  • If a comment only contains opinion, and no statements of fact (e.g. "The kid should be electrocuted!"), rate the comment according to your opinion (value system) and knowledge on the subject.
  • If a comment contains statement of fact, but you only have an opinion (i.e. emotions resulting from your value system) on the subject, but no real knowledge, do not rate the comment.
  • If a comment makes a point of which you are aware, which is true to your knowledge, but you deem it as little-known enough to warrant further exposure, rate the comment up. For the difference between and 4 and 5, you can use such factors as presentation, emotional agreement and minor errors as guidance.
  • If a comment informs you about something you didn't know previously, and you see it as sufficiently backed up by solid facts, rate the comment up.
  • If a comment makes a point which makes you think, but you do not know whether it is true or not, do not rate the comment.
  • If a comment makes a point which makes you feel bad, but you would not be able to explain why if the author inquired why you rated his comment a 1 or a 2, do not rate the comment.
  • If a comment contains false logic, lies or errors or otherwise undesirable content, and you could elaborate further on that if that became necessary (but do not have the time nor the will to do so), rate the comment down.

Generally, low ratings are frustrating and discouraging. That's the real and only reason people complain about them, the seemingly rational reasons they give for this behavior are, as we have found, illogical and inconsistent. Consider their emotional impact when you give low ratings and use them especially where you think the subject is of such importance that lies and disinformation should not be spread.

Of course, replying is always most desirable, but there are cases when the point in question has already been made in several previous comments and the comment in question has simply ignored them, or when you don't have the time to write a detailed reply (the most common reason to rate instead of replying).

In the long term, the rating system must evolve in order to reduce the amount of noise that it will inevitably continue to produce and in order to increase its usefulness. Reducing the emotional frustration caused by low comment rating by building a system that aggregates scores should be a primary goal - this would preserve the usefulness of the ratnig system, as comments with low total scores would be equivalent to those with low ratings, but not as frustrating. Eventually, a more personalizable system of user groups and trust would be preferable.

Sponsors

Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure

Login

Poll
If this story was a comment, I would rate it a ..
o 0 - meta noise is worse than spam! 30%
o 1 - because you're SO wrong! 5%
o 1 - because your story is so badly written 1%
o 3 - just your run of the mill meta story 24%
o 4 - some good points, but you invoked Godwin's Law! 12%
o 5 - because I agree with you 13%
o 5 - because it's well written, but I disagree 6%
o 1 - your editorial comment should have been topical 7%

Votes: 98
Results | Other Polls

Related Links
o FAQ
o refutation
o Also by Eloquence


Display: Sort:
The Trouble With Ratings | 76 comments (36 topical, 40 editorial, 0 hidden)
So basically... (4.23 / 17) (#2)
by codepoet on Tue Aug 07, 2001 at 04:48:02 PM EST

This is your description of why you rate everything as 1 or 5, but mostly 1. There are shorter ways of doing this.

Know drunk you are when Yoda you sound like, hmm?
how to rate (3.71 / 7) (#5)
by speek on Tue Aug 07, 2001 at 05:01:19 PM EST

No matter what the FAQ says, and no matter what anyone else says, the meaning of a rating is determined by the actions of the Scoop code. Since ratings are used to sort posts, then someone rating one post as a '3' and another as a '4' means they think the '4' post deserves to be seen before the '3' post (or, if time is an element, they would judge the '4' post as being more important to read before time runs out than the '3' post).

The other action of the code is to edit out posts with ratings below '1'. So, a '0' rating means that you don't think the post is worth reading, ever - even if the reader had infinite time to peruse K5.

Any other normative or qualitative description about what different ratings mean, or why to rate this way or that way simply represent that person's opinion on how they like to vote. I vote my way, and ignore them :-)

--
al queda is kicking themsleves for not knowing about the levees

dead horses (3.66 / 9) (#8)
by dr k on Tue Aug 07, 2001 at 05:19:46 PM EST

I have two favorite dead horses on this issue:

First, you can't mandate how people will rate comments, so to avoid the various inequalities of all the different methods, there should be a smaller range of ratings, like 1-3. Individual votes should have a minimal effect, because we are looking for the collective effect.

Second, we should consider the democratic implication of ratings. Highly rated comments are popular and agreeable. Mid level comments may be less popular, or they may be highly contentious. One is diminished by only reading the popular opinions.


Destroy all trusted users!

increase rating contribution, or junk ratings? (4.50 / 8) (#12)
by maynard on Tue Aug 07, 2001 at 05:30:06 PM EST

From one perspective, there's so much signal here on K5, why bother trying to sort the mess out? Why do we need ratings given the high quality of content overall? I could accept this solution.

However, if the rating system remains I think trying to impose a set of "objective" rules on a system which by definition is designed to elicit a value judgment from the readerbase is simply silly. Just drop the issue and let people rate as they will. That we now know who rates is enough to determine bias and that's that.

The more complex you make this the less people will be willing to rate, thus diminishing their value to readers. Add more drop boxes, meaningfull words like "informative", "insigntful" "Troll" etc (ala /.) along with the numerical value, and people will simply not participate because rating becomes too much of a pain in the ass. It's hard enough to get folks to rate as it is. Hell, even if someone rates you a "1" you ought to be glad that they even bothered to rate at all. This is why I think the arguments over how folks rate sidestep the real issue of low participation within the rating system.

Rusty is fond of saying (paraphrase) that a comment rating increases in value the more ratings the comment receives, which is why a single "1" has little effect on overall mojo while several "1"s slam your mojo hard. The implication here is that the greater the sample data the more valuable the rating overall. So be glad if someone thinks what you write is a bunch of shit and rates accordingly; at least they bothered telling you.

Ratings also reduce unnecessary comment clutter. There are a few here who expect written comments in reply justifying a low rating so as to further debate. However, sometimes it's not warranted or necessary. Your rating makes a statement in and of itself, thus removing the unnecessary clutter of a string of meaningless comments. Sometimes a reply simply adds noise and not signal, in this case a rating allows one to make a statement without actually entering a full comment.

I used to believe that ratings should only be done under strict rules of factual discourse. After a long debate with Rusty and Karsten, they convinced me that they're right -- by attempting to root out personal bias through adding code complexity one only reduces user participation. While the current system encourages bias, it also encourages contribution. Since a single "1" doesn't strongly effect total mojo, and since an unfair single "1" (unfair by majority community standard) tends to get a "5" in response from someone else, it all washes out in the end. Instead of arguing for stricter rules which can never be enforced, you should instead be arguing for wider rating participation -- or an end to the rating system itself. Because you will never code a utopia rating system which removes personal bias -- that's NOT a problem with an algorithmic solution.

JMO
--Maynard

Read The Proxies, a short crime thriller.

i mostly ignore them (3.25 / 8) (#14)
by beleriand on Tue Aug 07, 2001 at 05:32:34 PM EST

I think the 0 rating to get rid of spam is a good thing.

But i try to read most comments anyway.. if there not too many. And somehow i prefer to get the comments in chronological order.

Bad comments are easily skiped over, the brain gets really fast at this with training. If theres a long comment/story i often only read a few lines of it and just quickly scan through the rest.

Rating System Works Just Fine (4.00 / 5) (#30)
by RadiantMatrix on Tue Aug 07, 2001 at 06:53:02 PM EST

I will agree that the FAQ's postulate of rating solely on presentation is probably not the best of systems. However, I think the real point being made by the FAQ is that comments shouldn't be rated on whether you agree or disagree, but on the quality of the comment. Comments with severe factual and/or logical problems are low-quality.

I think people who refrain from voting based on agreement/disagreement and remember that factually or logically flawed comments are poor in quality do a lot to balance the ratings system. Also, keeping in mind that:

  1. is for "noise" or "spam" type comments that add nothing to the discussion
  2. is for comments that contribute but are poorly presented (some minor errors, badly written, etc)
  3. is for the "average" comment
  4. is for comments that are both well-presented and contribute to the discussion
  5. is for outstanding comments that are well-presented, contribute, and make excellent points
This is merely my re-statement of the descriptions in the FAQ.

I guess if I had to choose one piece of advice to raters to make the rating system more useful it would be to avoid the '1' and '5' extremes, saving them for the truly poor or truly outstanding comments, respectively. Overall, though the ratings system works well.

--
No amount of genius can overcome a preoccupation with detail.

Based on faulty assumptions (4.50 / 10) (#32)
by rusty on Tue Aug 07, 2001 at 07:06:00 PM EST

This story is a logical extrapolation of two faulty assumptions:
  1. The FAQ is The Word and should be adhered to religiously.

    This isn't true. The FAQ is a set of hints, suggestions, and approximations. Like the internet itself, this site operates mostly by "rough consensus and running code" with the emphasis on consensus. If every one of you decided that comments should be rated solely on how well they incorporate the word "agrarian", that would be how comments are rated, no matter what the FAQ says.

    The FAQ is there for people who desire guidance, not to be a rule book. It's the admins collective best guess as to what would be beneficial for the community. We may be wrong.

  2. The FAQ advocates ignoring truth.

    This is also incorrect. The bit about "agreeing" with the comment is much more intended to advocate reading things based on their own merits as an argument, not ignoring facts. The example you cite above would, as you say, be very easily countered, and thus is a poorly presented argument. Just because it's well-written doesn't make it not silly crap. I would assume K5ers are smart enough to not confuse the two. Part of presenting an argument is making it resistant to easy disproof. You can write like a dervish, but if you're arguing that the sky is usually green, you are presenting a poor argument.

    Besides all that, I am perfectly aware that people do rate based partially on agreement, and frankly, I think that's ok. If you want guidance, and are reading the FAQ, I'll encourage you to try to be as "objective" as possible in your ratings. But if you want to rate based on agreement, nothing's going to stop you. I think both can be factored together into this nebulous "quality" metric we're trying to obtain.

Basically, I hate to think the FAQ is being viewed as some kind of rule book. It's not. Take what you can from it, and follow your own sense of what's right. I'm not going to lay down a bunch of laws for you to follow.

By the way, yes I'm voting against this article, mainly because I don't think it's an original discussion, or particularly useful for the site. But I did think you deserved a response.

____
Not the real rusty

Misunderstandings (3.80 / 5) (#38)
by Eloquence on Tue Aug 07, 2001 at 08:21:51 PM EST

The FAQ is The Word and should be adhered to religiously.

I actually never made that assumption. My whole article would be blasphemy if I did, wouldn't it? :) I would expect your view on the FAQ to be what it is, and with that in mind I wrote the article. Keep in mind, however, that it is referred to as a set of guidelines in the case of rating disputes, and it should therefore accurately reflect the opinion of the users.

This is also incorrect. The bit about "agreeing" with the comment is much more intended to advocate reading things based on their own merits as an argument, not ignoring facts.

Um, yes, that's the kind of counter-argument that always amazes me in these discussions. "When I say 'don't rate based on agreement' I actually mean 'rate based on whether the comment is factually correct'." If I think that the comment is factually correct, then I most likely agree with it, don't I? I have explicitly discouraged rating purely on the basis of emotions and suggested rating on the basis of perceived knowledge. This is, however, still rating essentially on the basis of agreement.

Do you now see that the FAQ is, in essence, contradictory?
--
Copyright law is bad: infoAnarchy Pleasure is good: Origins of Violence
spread the word!
[ Parent ]

How about... (4.16 / 6) (#43)
by rusty on Tue Aug 07, 2001 at 08:56:39 PM EST

If you think the FAQ is bad, write me a section to replace what's there, and we'll work together on it, to make it clearer and more sensible. Email rusty@kuro5hin.org, I'd be glad for the help.

____
Not the real rusty
[ Parent ]
I agree with Eloquence (3.00 / 3) (#45)
by ToastyKen on Tue Aug 07, 2001 at 09:47:14 PM EST

I found this to be an excellent article because it very much reflects reality. I think the FAQ is seen as what you should do, and given its official nature, I would definitely say that people take it as "gospel" to some degree; we just don't necessarily follow it anyway. Like many things in life, reality does not match the "rulebook" here, and I think that's generally a bad thing, just as all those silly laws that are still on the books are generally bad. Why have the FAQ written in such a way that it only states roughly what should be, when it could state what really should be?

Anyway, I would be all for Elo's collaboration with Rusty to improve the FAQ, as I think that's the goal of this article anyway.

[ Parent ]

Not Contradictory: (3.33 / 3) (#62)
by Elkor on Wed Aug 08, 2001 at 11:23:23 AM EST

Potential Fact:
"Federal Subsidies of $$$$$$/year prevent farms from going bankrupt."

The above statment is a plausible fact. But, I don't know that it is a fact (i.e. can provide corroberating evidence to support the statement).

Further, I disagree with the concept of paying people not to do things. (Paying farmers to not grow crops to prevent a glut in the market, paying welfare to able bodied individuals capable of working but who have manipulated the system in their favor, Federal government employees in general, etc).

Now, depending on the conversation, this can either be a valid point (worth a 3 if the discussion is about federal subsidies) or a good argument to a stated position (worth a 4 if the argument is about economic prosperity/failings of small farmers).

But, I still disagree with the concept behind the fact.

Regards,
Elkor


"I won't tell you how to love God if you don't tell me how to love myself."
-Margo Eve
[ Parent ]
In this case (3.33 / 3) (#63)
by Eloquence on Wed Aug 08, 2001 at 11:51:48 AM EST

I don't know that it is a fact (i.e. can provide corroberating evidence to support the statement)

In this case you should, IMHO, abstain from rating the comment.
--
Copyright law is bad: infoAnarchy Pleasure is good: Origins of Violence
spread the word!
[ Parent ]

But the sky is green! (2.00 / 1) (#60)
by wiredog on Wed Aug 08, 2001 at 09:03:34 AM EST

Or maybe it's those new yellow lenses in my glasses...

If there's a choice between performance and ease of use, Linux will go for performance every time. -- Jerry Pournelle
[ Parent ]
Once a day there's a green flash at sunset! (2.00 / 1) (#67)
by maynard on Wed Aug 08, 2001 at 10:21:36 PM EST

Does this count? *cough*! --M

Read The Proxies, a short crime thriller.
[ Parent ]
Ratings (3.00 / 6) (#33)
by Weezul on Tue Aug 07, 2001 at 07:06:02 PM EST

Ratings attempt to represent many diffrent types of value with one single number. You should not expect them to represent your perspective on "value." If you want a system which will attempt to make people rate things "by the rues" try the follwing:

Allow four (say) yes/no ratings: well writen, informative, funny, and correct. Allow everyone to specify the number of "top rated" comments they will read, i.e. I want to read the 2 funniest comments, the three most informative comments, the one most well writen comment, and none of the correct comments. Now, I will read exactly 6 comments from every article (unless I say want more of something for a specific article).

Now, voting is much more clearly defined by "well writen, informative, funny and (subjectivly) correct. This means that there is a more clear problem when people vote stupid shit as informative or drible as well writen, but people will feal free to rate "correct" out the ass to express themselves.. and no one will notice.

Hell, you might as well just go and premote these 6 comments to my version of the front page as edditors notes or addendums. That will get people pissed off when stupid comments are rated too high. Plus, it will reduce the total number of comments people post, always a good thing, since most readers will never see your comment unless it's really good.

"Fascism should more appropriately be called Corporatism because it is a merger of state and corporate power." - Benito Mussolini
If it ain't broke... (4.14 / 7) (#35)
by anansi on Tue Aug 07, 2001 at 07:23:58 PM EST

tinker with it until people can't figure out what it does any more!

The FAQ is not gospel: If an author pushes my buttons in an interesting way, I'll rate it up, If they sling insults, I rate them down. If their comment is confusing or unclear, I'll ignore it.

The only changes I can imagine that would improve feedback are:

1) Make it so I can see my own hidden comments. I've had comments vanish, and I wish I could look back and remind myself what not to say in the future.

2) Generate a total summary of my rating habits: Of the comments I bother to rate, what pecentage get a 1, 2, 3, etc.?

3) What's my average word count when I post a comment?

4) What's my Mojo? I wouldn't want this to be public knowlege, but it would be nice to keep track of for persoanl curiosity.

Don't call it Fascism. Use Musollini's term: "Corporatism"

summary of my rating habits (4.00 / 3) (#50)
by codemonkey_uk on Wed Aug 08, 2001 at 04:52:28 AM EST

The other site recently followed kuro5hin's lead and introduced full disclosure to its rating system (interestingly, they use a two dimensional rating system). In addition they added average rating stats to the user pages. This is surprisingly useful, as some people have much higher standards than other people, so if you get a 1 or a 2 for a photo I've posted, and check out the user page for the person who rated me, I can see that if they hand out on average 3/10, they simply have very high standards, where as if they hand out 9's and 10's like candy, they really think I suck.

I think this is a *good* thing and is definitly worth considering here.

And extension of this idea, is to modify the rating given, by the user's average rating, making a 5 from someone who averages 1 or 2 more "effective" than a 5 JCB (and vice versa). :)
---
Thad
"The most savage controversies are those about matters as to which there is no good evidence either way." - Bertrand Russell
[ Parent ]

you're missing something... (5.00 / 1) (#58)
by gibichung on Wed Aug 08, 2001 at 07:34:16 AM EST

I tend to give high ratings to people who I think should be heard, and reply to those I disagree with. The problem is, after a few rounds of discussion, I believe it's inappropriate to rate the people you've been arguing with. This develops into not wanting to rate bad comments that you disagree with: are you doing it because it's genuinely bad, or just that you disagree?

That being said, I don't think any change in the rating system is required. People with strong opinions about things are always going to be less impressed with the arguements of the other side. Disclosing which users make the ratings keeps this in check, and it works pretty well.

-----
"No man is above the law and no man is below it; nor do we ask any man's permission when we require him to obey it." -- Theodore Roosevelt
[ Parent ]

2 votes and ranking (3.00 / 4) (#37)
by mrBlond on Tue Aug 07, 2001 at 07:50:52 PM EST

A meta vote and an opinion vote and rated rank

"The reasonable man adapts himself to the world; the unreasonable one persists to adapt the world to himself. Therefore all progress depends on the unreasonable man." - George Bernard Shaw
--
Inoshiro for cabal leader.

No (4.00 / 5) (#47)
by qpt on Wed Aug 08, 2001 at 01:02:20 AM EST

The aforementioned "emotional frustration" is partially intended since it discourages spammers, trolls and flamers.
I suppose one out of three is not that bad, but I expect more! Generally, nobody trolls or flames for the purpose of being moderated well. There is a subset of positive moderation trolls, but they are in the minority and are not really worth discussing anyway. People troll for a response and they flame to let off steam.
There is no absolute truth in human knowledge, but there are approximations to the truth.
Is that true? Or is it just an approximation of the truth? If it is not strictly true, does that mean that there are absolute truths in human knowledge, thus rendering the statement false? If the assertion does not have any exception at all, what keeps it from being absolutely true, thus making it false? Quite a dilemma presents itself!

Perhaps you are inhuman and are thust capable of knowing the absolute. However, you should realize that most of Kuro5hin's readership is human. Please go easy on us.

Glibness aside, you appear to have ingeniously affirmed a contradiction in an article encouraging us to consider truth. I am having trouble deciding if you are clever and ironic or mind-blowingly stupid. Brilliantly, I cannot even vote the story down for being false, since that is precisely what the story suggests and following false advice would be irrational on my part. However, it would also be irrational to vote it up, since it is false.

Oh bother, your little logical circus has exhausted both my patience and reason.

Domine Deus, creator coeli et terrae respice humilitatem nostram.

Why rating noise exists (4.33 / 9) (#48)
by Sunir on Wed Aug 08, 2001 at 01:46:08 AM EST

I've been working on a presentation on soft security which included a discussion of how hidden comments work on Kuro5hin. In the midst of this, I lucked upon an interesting case where a hidden comment was unhidden despite the original author asking for it to be moderated to 0. Worse, because it was continuously bumped above 1, people heaped 0 upon 0 onto the hapless author, lowering his Mojo.

The failure was that if you follow the links from Review Hidden Comments, the page you are directed to has "mode=alone" in the URL, thus masking out replies to that post. This masks context.

Indeed, the failure of all numeric rating systems is the lack of context. We can loosely summarize context by answering the five W questions, Who? What? Where? When? Why? (we all know How? in this case--rate, click the button).

What? is easy: it's the rating. Where? is also easy: it's the comment being rated. Who? used to be a little more dicey. Originally, the names of the raters were unknown. Due to some prompting, Scoop was changed to reveal the raters' identities. This revealed Who?

However, When? is unknown. Worse, Scoop orders the authors completely randomly, so it is unclear even in what order people rated each other. And Why? is completely unknown. In the anecdote I mentioned above, Why? was easy: just read the author's own reply. But the saviours missed that because they had no context. In the general case, the Why? is completely missing. This leads to the "rating noise". Lack of information is the number one cause of conflicts.

Unfortunately, carefully presenting why you rated a comment low to an author is dangerous to your Mojo. The first time I told an author why I rated his post a 1, I received a positive response, even noticing the author mentioned the event in a comment later on as an example of good behaviour (cool!). The last time I did it, I received a massive flame back and many low ratings for several comments I had recently made. This convinced me not to change my rating.

I just shrug my shoulders. People take the rating system too seriously. One bad rating isn't going to kill you. Others will average that out. And often the rating doesn't mean anything, especially deep in a thread, because people are going to be reading the entire thread anyway. In the grand scheme of things, your Mojo isn't affected either, so for those who childishly cherish their Trusted Moderator Status, you shouldn't be worried.

The real value of ratings is to remove spam in a stable, "no false positives" way. If you review hidden comments, you'll notice all of the hidden comments deserve their ranking. To this date, no one has ever been banned from Kuro5hin due to low Mojo. Those who have Trusted Moderator Status are actually in a worse position because they get to read all the spam. I don't think you should be concerned overly about maintaining that rating.

But, if you are, a long time ago I discovered how to defeat the system. Don't post crap. I asked Rusty to look into fixing this loophole and he assured me he would work on it. In the meantime, feel free to exploit this glaring security hole for your own nefarious ends.

"Look! You're free! Go, and be free!" and everyone hated it for that. --r

Trusted user (3.25 / 4) (#53)
by pallex on Wed Aug 08, 2001 at 05:14:28 AM EST

Surely a solution would be for all users to have the appearance of being a trusted user (and so have `0` as an option for moderating) but only people who are actually trusted (this wouldnt be visible, of course) would have an effect in the database?


[ Parent ]
Violates open process; entirely defeats Mojo (4.50 / 4) (#68)
by Sunir on Wed Aug 08, 2001 at 10:27:25 PM EST

This violates open process, which means it will result in people becoming aggrieved that their rating did not actually affect the outcome. They might assume underhandedness on the part of the system administrators. See The Case of Badvogato to understand why magic algorithms suck.

Worse, because you would have to present a fake rating for the users who are not trusted (because they can equally calculate the average), weird results may occur. Consider a post that is just on the line between hidden and invisible. An untrusted user rates it zero, thus "hiding" it in her view. However, she observes that others continue to reply to the post. From this, she might conclude that either trusted users have taken the habit of replying to spam, or that she is not trusted.

Much worse, since only trusted users can see hidden comments currently, you will have to break this feature to allow everyone to see hidden comments--thus annhilating the feature altogether. This will also circumvent the entire purpose of hiding comments: to kill the "spam club". That is, if you cannot see your own spam, and you cannot see others' spam, there is no chance you will create a spam club. There is no reward for "tagging" the site. The alternative is Slashdot's broken model, to which Mojo was created in reaction.

"Look! You're free! Go, and be free!" and everyone hated it for that. --r
[ Parent ]

Yes, How about Truth? (4.00 / 5) (#49)
by ti dave on Wed Aug 08, 2001 at 04:11:21 AM EST

"In this case, almost everyone knows that the comment is erroneous and would be able to easily present a refutation."

What about comments that purport to be true, concerning less obvious topics?
I would encourage every k5 reader to do their own investigations and determine the veracity of comments they read here. Don't always have information spoon-fed to you.

"By rating the comment up, however, those without a clue could easily be fooled by the rating into believing its content, especially if there is no reply to it."

See above.

"Presentation should obviously not be any criterion for rating, exactly the opposite is true!"

How so?
Do you mean that "lack of presentation" or "poor presentation" merits a higher rating?

For me, k5 isn't a matter of style over substance, nor substance over style. It's about having the right balance of those qualities.

Provide both, and I'll rate you well.

"The FAQ itself suggests rating up comments which make a "good point" -- what is a good point if not a post that is logical and true to our knowledge?"

To me, and I don't believe I'm being extreme on this, a "good point" *can* be a point that I don't necessarily hold true at the moment I read it.

If the "point" causes me to re-evaluate my position on a topic, or shows me that my previously held opinions have flaws, the it can be a "good point".

I enjoy having my assumptions challenged. I enjoy learning new things, and I'm not offended when people suggest their positions may be better than mine.

I'll use my reasoning abilities to investigate further, and try to keep an open mind while I evaluate the new information.

Lastly, I take pride in the fact that I've spent most of my life trying to acquire as broad a base of knowledge as I can. I have encountered very few topics here that I had less than fundamental awareness of. I don't rate comments down if I don't feel I have a grasp of the topic.

Cheers,

ti dave
"If you dial," Iran said, eyes open and watching, "for greater venom, then I'll dial the same."

The basis of ratings problems... (3.33 / 3) (#54)
by simon farnz on Wed Aug 08, 2001 at 06:35:28 AM EST

... is that people get worked up about them. By all means point to the FAQ/your favourite scheme to give people an idea of how to rate; that is just part of the discussion. Do not get upset if people rate you down when you feel you shouldn't be, or refuse to rate according to your favourite scheme. Don't worry about your trusted user status either; sure it is a nice honour for some people, but you don't lose much if you don't have it.

I admit to not liking to rate people down, and to be idiosyncratic in my rating style, and to feeling upset when a comment I liked is rated down, when one I thought was tosh is rated up, but it is not important in the k5 sphere, let alone the real world.
--
If guns are outlawed, only outlaws have guns

Well (4.00 / 5) (#55)
by AmberEyes on Wed Aug 08, 2001 at 06:44:18 AM EST

"A comment which presents some little-known truth, even if not well-written, should be rated to the top, while revisionist bullshit like the one quoted above should quickly fall to the bottom where it belongs."

Right. So, instead of marking it up so that it is in attention and CAN be debunked by someone more knowledgable, you would rather it fall "through the cracks" and be ignored, so that more misinformed people would continue spreading around the rumors should they happen to run across it?

I frequently vote stuff up (especially when I rate submissions) that I think are particularily stupid or silly, just so it has a better chance of being seen and debunked, or so people can explain to the poster how silly it is.

I voted this story to +1 section, however, I think that the rating system is fine as is, and we don't need to change it. Why? Because I want discussion so that either it can be proved that your ideas aren't necessary, or so that I can skim off some idea that is beneficial in regards to your rating system, but that wasn't made immediately apparent in your post.

Your system seems to be "If you don't like it, or it's a lie, then bury it."

My system is "If you don't like it, or it's a lie, then show it to be wrong so it's not repeated."

Different strokes for different folks though. :)

-AmberEyes


"But you [AmberEyes] have never admitted defeat your entire life, so why should you start now. It seems the only perfect human being since Jesus Christ himself is in our presence." -my Uncle Dean
No censorship, just ranking (3.33 / 3) (#56)
by Eloquence on Wed Aug 08, 2001 at 07:20:55 AM EST

Right. So, instead of marking it up so that it is in attention and CAN be debunked by someone more knowledgable, you would rather it fall "through the cracks" and be ignored, so that more misinformed people would continue spreading around the rumors should they happen to run across it?

The idea is not to remove the post or make it invisible. A zero rating would be inappropriate. The goal is to highlight the more important posts on a subject while still leaving room for debunking for people who have a very broad interest in the subject. You see, if you are open-minded and want to hear some radical opinions you haven't heard before, you will seek out these opinions regardless of their rating. This is not possible if opinions are effectively censored, therefore it shouldn't be done. But the ranking helps those who quickly want to get to the "meat" of a subject.
--
Copyright law is bad: infoAnarchy Pleasure is good: Origins of Violence
spread the word!
[ Parent ]

A bit clearer (3.00 / 1) (#59)
by AmberEyes on Wed Aug 08, 2001 at 08:35:28 AM EST

Thanks for making that a bit clearer. I was certainly hoping you were not advocating censorship.

I'm still worried that a "mod stupid things down" attitude would be counter-productive to helping remove those radical opinions that are wrong....you know and I know that the Nazis gassed the Jews during the Holocaust, and maybe we chuckle and grin over the nonsense of someone denying that it happened. We can actively seek those posts out, yes, but what about someone who stumbles onto something, and for one reason or another, decides to take it as gospel. The lower the mod rating a message has, has shown to me in the past that it gets fewer replies. Rating an insane comment like "Racism is good because it strengthens individual cultural stances by creating massive scisms within populations" or "Poor people obviously don't work as hard at getting a good job as rich people do" simply makes them less likely to be responded to..and more likely to be picked up as gospel by a radical with his head screwed on too tight. :)

In reality, though, you can't fix it, because how do you then determine the importance of modding something up or modding something down if it's "for the good of the many"? No rating system is perfect, or free from abuse, which is why I don't think we need to change the one we have. It seems to work most of the time, and on the occassions that it doesn't, it's usually corrected pretty quickly by a Good Samaritan voter. In fact, if it was up to me, we'd do away with rating systems altogether (except for the submission page). But that's just me. Heh, and it would piss a lot of people off, I'm sure. :)

Don't think I'm slamming your (as in, only your) idea, per se. I'm slamming the idea of simply changing it. It seems a lot more trouble than it's worth. :)

-AmberEyes


"But you [AmberEyes] have never admitted defeat your entire life, so why should you start now. It seems the only perfect human being since Jesus Christ himself is in our presence." -my Uncle Dean
[ Parent ]
Correlation, not consequence (4.00 / 2) (#66)
by Sunir on Wed Aug 08, 2001 at 10:19:50 PM EST

The lower the mod rating a message has, has shown to me in the past that it gets fewer replies.

That's a correlation, but it isn't necessarily a consequence. A low rated comment may get fewer replies not because of its rating, but simply because it is crap. Who wants to reply to crap? Conversely, highly rated comments are often strongly stated, and thus strongly debatable, and thus collect replies.

It's not at all obvious whether in the current state of Kuro5hin a significant proportion of readers and--more importantly--writers actually take into account ratings when using the site. It's quite possible that many people are like me in that we read all the comments regardless of their rating. Thus, rating a bad comment up will not succeed in attracting more refuting replies, but it will succeed in sending a message to the author of the bad comment, "Good work!" This will give a crossed message, not what you intend.

"Look! You're free! Go, and be free!" and everyone hated it for that. --r
[ Parent ]

I disagree (3.66 / 3) (#61)
by Simon Kinahan on Wed Aug 08, 2001 at 10:48:45 AM EST

To pick a potentially relevant nit: you say the text you quote from Zundel's website is factually incorrect, but in fact it makes no false statements. It only refers to false statements (by Leuchter and Zundel themselves).

Regardless of that, I would, in fact, rate such an comment up if it had no reply, and I did not intend to refute it myself, and leave it alone otherwise. Why ? Because people who pass of lies on the basis on spurious "authority" thrive in darkness, and tend to show themselves up as fools and liars only when exposed to light.

Given that most people have never heard of holocaust revisionism, and are not familiar with the forensic and documentary evidence that the holocaust occurred, and are therefore likely to "see both sides of the argument" if presented only with the dissenting side, it is far, far, more effective to rate such things up in the hope that someone knowledgeable will reply, or better still to reply yourself, than to rate the comment down. If it gets rated down, it looks like an attempt to supress dissent, and a certain kind of slightly paranoid person will assume "there must be something to it". The same goes for creationists, people who believe HIV is not the cause of AIDS, and most other conspiracy theories. Its only the fact that even the educated public is fairly ignorant that makes these people look credible.

What is more, in most K5 discussions the truth and falsehood of claims are not nearly as clearcut, which is why some discussions keep coming up. We've only had one "HIV is not the cause of AIDS" article, because it got voted up and then decisively stomped on, but the "Globalisation is an Evil Capitalist Conspiracy" thing comes up again and again, because there is almost no decisive on either side that will convince the other.

If you check the ratings of comments in the globalisation articles, you'll see that comments get high ratings from one group of people and low ratings from another group, or vice versa, based on political agreement or disagreement, regardless of whether or not the articles are correct. Its in these cases that rating by agreement shows it worst effects, to the extent that people accuse one another of trolling and dish out very low ratings, even for clear and original commentary, based on "agreement".





Simon

If you disagree, post, don't moderate
Update (4.50 / 2) (#65)
by Eloquence on Wed Aug 08, 2001 at 07:18:57 PM EST

I have sent a suggestion for changing the FAQ section on comment ratings to Rusty, which he has now kindly included. I have tried to incorporate some of the ideas from the discussion as well and to leave the description as open to individual interpretation as possible without being unspecific. I'd be happy to get input on the result.
--
Copyright law is bad: infoAnarchy Pleasure is good: Origins of Violence
spread the word!
good job. (3.00 / 1) (#72)
by garlic on Thu Aug 09, 2001 at 06:58:40 PM EST

good write up. Definately better than what was there before.

HUSI challenge: post 4 troll diaries on husi without being outed as a Kuron, or having the diaries deleted or moved by admins.
[ Parent ]

Make your depth two hundred meters. (4.00 / 1) (#73)
by kitten on Mon Aug 20, 2001 at 06:11:40 AM EST

I generally don't pay that much attention to ratings, especially the ratings on my own comments. I just don't care that much.

Regardless, I'll use one of my own comments as an example. Here, the comment was rated an average of 4.45 by 20 people. 19 of these people rated it 4 or 5, except for one person who rated it 1.

Certanly this doesn't bother me overmuch, but I do find it interesting that the comment was rated so low by one person for no apparent reason other than he disagreed with me.

There are several people that I could name but won't who are notorious for rating well-written, well-reasoned comments ridiculously low for no other reason than the view expressed conflicts with their own. Contrawise, these same people have been observed to grant high ratings to comments which are almost universally agreed to be garbage - simply because the author of the comment happened to agree with them.

I don't do much comment-rating myself; I prefer to reply directly. However, when I do rate, I try to do it based on
  • Style: Is this well-written or not? Is the author making a point, or is he just lashing out and being vicious? How's the structure - is it coherent? Does the post flow from point A to B to C, or does it just sort of ramble without ever trying to reach a conclusion?
  • Substance: Does this guy know what he's talking about, or is he pulling information from his ass? Are his facts correct? If he's making claims that seem a bit strange, are there citations or links? Does he support his point? Is he actually focusing on the issue at hand, or is he off on some tangent or unrelated topic?

    If I'm in a comment-rating mood, I'll rate a comment highly if it's well-written and has something to say, even if I disagree with it. If it has something to say but appears to have been written by a five-year-old with Down's syndrome, I'm rating it down. If it's written with prose that would make Shakespeare jealous, but has absolutely no point to it, I'll rate it down.

    I really don't see what's so hard about this. I don't see why other people demonstrate such immaturity with their ratings. I also don't see what the big deal is - while I do like to see my comments rated up, they aren't that big a deal.

    mirrorshades radio - darkwave, synthpop, industrial, futurepop.
  • My perspective as a newcomer (4.00 / 1) (#74)
    by Defiant One on Wed Aug 22, 2001 at 03:13:51 PM EST

    I agree on many points with your paper. In fact, I'll chime in to the effect that the ratings are at best a useless attempt at moderation. It also might wrongly teach people that popularity equates with rationality.

    My perspective is this: I only noticed Kuro5hin a few months ago, when I got a few hits on one of my personal website pages which someone had linked in a commentary on Objectivism. I had no idea this site existed before then, and I wasn't a poster on Slashdot - it had only come up during Googling for other info. I used to post heavily on Usenet, in the philosophy groups, and what I both liked and hated about it was the complete lack of moderation or supervising authority. While anarchy can be frustrating when dealing with a troll, it can be wonderful in the middle of a well-reasoned, fast, online discussion.

    So, when I checked out Kuro5hin because of the link to me, and saw how slowly the stories and comments were posted (compared to Usenet or even Slashdot, it's quite a boring pace), and then all these comment ratings like on Slashdot, and I thought, "Oh, it's another circle jerk". It reminded me of Woody Allen's comment about intellectuals being like the Mafia, in that "they only kill their own kind".

    But seriously, reading Kuro5shin and Slashdot can sometimes be a little like a clique, because of those unnecessary ratings. I mean, who really cares if a comment is a troll, or if another is a logical treatise? Isn't the whole stated point of these discussion forum sites to foster discussion? If my comments are going to be rated down, or not rated at all, because you folks don't like what I have to say, then I might as well stay offline and go to a Rotary Club meeting! Better yet, I might just go back to Usenet.

    One last note: Though I like the design, I am not a subscriber to this site, and I view it via a free Webwasher Proxy, so I don't see any adverts or popups. I think this is a good thing, and partially comes from my time on Usenet using an email program. I also, for servers, use Redhat on custom built hardware and Solaris on Sparc at the ISP, so VA Linux (the obvious sponsor of K5 and '/.') isn't making anything off of me or my company, and if my comments are rated down, even the benefit of my words is lost, which again, for a discussion forum is ironic. I can foresee a day when I will drift away from this site and Slashdot too, in favour of the next discussion site or whatever, never giving a penny of revenue to VA or it's advertisers.

    So, to boost the discussions, and retain people with perspectives such as mine, my advice is to get rid of the rating system, but for the polls and submission votes, something this site has as an advantage over Slashdot. There is virually no benefit in the comment ratings, and to claim they increase the quality of the discussions is merely to trick oneself into a justification for the clique as a moderator - something which has never been good for openminded thinking.


    "What can I say, I believe in total, honest democracy. I also believe this American system can work."
    - Woody Allen, Stardust Memories


    Participation without support (4.00 / 2) (#75)
    by maveness on Fri Aug 24, 2001 at 02:10:13 PM EST

    One last note: Though I like the design, I am not a subscriber to this site, and I view it via a free Webwasher Proxy, so I don't see any adverts or popups. I think this is a good thing, and partially comes from my time on Usenet using an email program. I also, for servers, use Redhat on custom built hardware and Solaris on Sparc at the ISP, so VA Linux (the obvious sponsor of K5 and '/.') isn't making anything off of me or my company, and if my comments are rated down, even the benefit of my words is lost, which again, for a discussion forum is ironic. I can foresee a day when I will drift away from this site and Slashdot too, in favour of the next discussion site or whatever, never giving a penny of revenue to VA or it's advertisers.
    I'm curious: why did you choose to emphasize this?

    I'm not a subscriber either, and I blithely ignore any ads that appear in my browser window, so I'm not coming from a holier-than-thou perspective. But I'm struck by the emphasis you place on this, it's almost like a conflict-of-interest disclaimer (e.g., "I'm SO not making a profit from any association with kuro5hin.org -- so much so that I won't even let them show me advertisement and possibly make a penny themselves or cause their sponsors to thrive in any way either!") It seems odd in the context of the rest of your comment.

    The only thing I could think of was that this was your way of establishing the purity of your 'netcred' (e.g., "Hey, I've been around a long time, I know how to customise a server, and I'm not in it for the money").

    Honestly, I'm not trying to flame you here, I'm just wondering what the motivation was. Because as I read it, it sounded like you were saying, "Your comment ratings system is stupid, I don't want you to ever make any money, and I'll be moving on any day now." Hardly a community-oriented sentiment (thanks for stopping by)!

    *********
    Latest fortune cookie: "The current year will bring you much happiness." As if.
    [ Parent ]

    it's about *their* 'netcred' (3.00 / 1) (#76)
    by Defiant One on Fri Aug 24, 2001 at 05:22:25 PM EST

    Good questions.

    "I am not a subscriber..."
    I'm curious: why did you choose to emphasize this?
    Well, it had nothing to do with my 'netcred' or a disclaimer. Basically it was just to say that I would not wish to pay K5 only to (1) have my words rated by people who may not want to listen, who may not be subscribers, and who may only be rating me for emotional reasons, and (2) gain the ability to turn off the ads, when I don't see them anyway, and since it seems silly to pay someone to turn off their automated adverts.

    So, the upshot is I would not wish to support these decisions by subscribing, especially given that the ratings will eventually drive me off anyway (even though I prefer K5 over Slash because of the wider subject matter, the polls, submission voting, and diaries). I've already seen this with Slashdot, where I have a karma of -2 after only a couple of weeks of posting. It happened just today, when I caught them in a conflict of interest regarding VA Linux's heavy financial losses, and my submission of that fact was rejected, I was rated down in response to similar submission because of it, all when someone could theoretically respond to me with a smartass post and get a "Score 5: Funny".

    I just think that's deplorable for that which purports to be a community, and it really destroys their credibility, not mine, because what I posted would have been fine if it involved any other company in the world but VA Linux. Since I see a VA Linux advert built into every page on K5, I fear this kind of thing may happen here too, or has happened already.

    This is why I mentioned Usenet, where in most groups, NOBODY can moderate you out or rate you down for such irrational reasons. A user can block your messages from their reader, but that's their individual decision.


    "What can I say, I believe in total, honest democracy. I also believe this American system can work."
    - Woody Allen, Stardust Memories


    [ Parent ]
    The Trouble With Ratings | 76 comments (36 topical, 40 editorial, 0 hidden)
    Display: Sort:

    kuro5hin.org

    [XML]
    All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
    See our legalese page for copyright policies. Please also read our Privacy Policy.
    Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
    Need some help? Email help@kuro5hin.org.
    My heart's the long stairs.

    Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!