Why do we need ratings anyway?
Ratings tend to lead to different kinds of noise in discussions. Complaints about bad ratings, reasons for these ratings, stories about the rating system etc. This kind of meta-noise is almost always annoying for the majority of participants in the site. Add to this the emotional frustration of receiving bad ratings and you have to answer the question why ratings should be used at all.
There are several reasons for the existence of ratings. The aforementioned "emotional frustration" is partially intended since it discourages spammers, trolls and flamers. With the zero rating effectively making a post invisible to its author, the point of posting insults, bad links or other undesired content, getting a wide reaction from the site's audience, is entirely lost -- even if there is a reaction, the poster will not be able to observe it without first gaining trusted user status with an additional account. So ratings combined with reputations are a very efficient protection against abuse.
Furthermore, ratings make it possible to sort posts according to their quality (we get to the definition of this term next). In a lot of stories, many comments will simply consist of verbalized emotions, half-knowledge or actual disinformation. Surely a general rating system will not make it possible to truthfully, objectively sort all comments, but it should give the reader certain reading priorities.
What is quality?
There is no absolute truth in human knowledge, but there are approximations to the truth. Those who see the world as entirely the result of purely subjective experience, a construct of our minds alone, fail to explain why some ideas seem historically to work better than others in improving the conditions of our lives. The method with which you access and read this article is part of this innovation through understanding. Objective quality -- which, again, is not and does not claim to be absolute -- exists, and through peer review we can try to find it.
Rating comments can be seen as essentially a voting process in which the readers decide how well the comments describe reality. If they have no prior opinion or knowledge on the subject at hand, they should abstain from rating comments since their rating would only reflect the post's presentation and not its content. It is hoped that the majority will be able to decide properly on which comments best describe reality, since on any given issue, it is assumed that the majority has the proper education to make this decision.
This is, obviously, a wrong assumption, at least in our current society. So a rating system which is based on majority rule will often have undesirable outcomes. People with no knowledge on a subject will still want to have their "votes counted", especially if they are convinced by a comment's presentation. A better rating system would be much more complex. It would include not only the rating of comments, but also the rating of persons. It would allow peer groups with specific interests to form and to select (and group) experts into teams which, together, select content of good quality (and persons to join them in their particular field of expertise).
But this is not the rating system we are using on K5, so we should make the best of what we have. A simple fix to some problems associated with the current system may be to change the rating in such a way that it aggregates votes into a total value, so that comments get a total score (with no upwards limit) and the best comments are "selected", rather than elected -- this would avoid the emotional frustration associated with bad ratings while maintaining the aspect of quality selection. It would not be as personalized as a trust-based rating system, but better than what we are using.
If we want to keep using what we are using, however, the voting process outlined above seems to be quite reasonable, as compared to the alternative, which is:
The Myth of Emotional Disagreement
The FAQ suggests exactly the opposite to the rating method described above. As of today, it says this (it was changed in the past and will likely be changed again):
"Try not to vote based on whether you AGREE with the comment or not, or whether you like the person posting the comment. Try to base your ratings on HOW the comment is presented. We understand that this is hard, and that most of us are human and let emotions come into play."
Let's say there is a story on the front page that discusses the Holocaust and restitutions related to slave labor. In this story, we find a comment that has the following content:
America's leading gas chamber expert, Boston engineer Fred A. Leuchter, carefully examined the supposed "gas chambers" in Poland and concluded that the Auschwitz gassing story is
absurd and technically impossible.
Now, according to the guidelines in the FAQ, this is a real gem of a comment, "pure gold" (let us ignore the fact that it was copied from Holocaust revisionist Ernst Zündel's website). Anyone who would rate such a comment down must be acting on their primitive emotions. It is eloquently written, comprehensive, includes sources and even buying information! It is a good counterpoint well-presented. What more could we ask for?
Leuchter is the foremost specialist on the design and installation of gas chambers used in the United States to execute convicted criminals. For example, he designed a gas chamber facility
for the Missouri state penitentiary.
In February 1988 he carried out a detailed onsite examination of the "gas chambers" at Auschwitz, Birkenau and Majdanek in Poland, which are either still standing or only partially in
ruins. In sworn testimony to a Toronto court and in a technical report, Leuchter described every aspect of his investigation.
He concluded by emphatically declaring that the alleged gassing facilities could not possibly have been used to kill people. Among other things, he pointed out that the so-called "gas
chambers" were not properly sealed or vented to kill human beings without also killing German camp personnel. (note 20)
Dr. William B. Lindsey, a research chemist employed for 33 years by the Dupont Corporation, likewise testified in a 1985 court case that the Auschwitz gassing story is technically
impossible. Based on a careful on-site examination of the "gas chambers" at Auschwitz, Birkenau and Majdanek, and on his years of experience, he declared: "I have come to the
conclusion that no one was willfully or purposefully killed with Zyklon B [hydrocyanic acid gas] in this manner. I consider it absolutely impossible."
The Leuchter Report: An Engineering Report on the Alleged Execution Gas Chambers at Auschwitz, Birkenau and Majdanek (Toronto: 1988). Available for $17.00, postpaid, from the IHR.
The Globe and Mail (Toronto), Feb. 12, 1985, p. M3
How about truth?
In this case, almost everyone knows that the comment is erroneous and would be able to easily present a refutation. By rating the comment up, however, those without a clue could easily be fooled by the rating into believing its content, especially if there is no reply to it. We can be quite sure that in this case, the comment would be quickly voted down, perhaps even zeroed (although some people, out of principle, would certainly fetch it up again by giving it a 5 if it crossed the threshold of invisibility).
Presentation should obviously not be any criterion for rating, exactly the opposite is true! The FAQ itself suggests rating up comments which make a "good point" -- what is a good point if not a post that is logical and true to our knowledge? The ridiculousness of the suggestion to rate based on presentation rather than content could not even be consistently kept up by its author through a few paragraphs.
A comment which presents some little-known truth, even if not well-written, should be rated to the top, while revisionist bullshit like the one quoted above should quickly fall to the bottom where it belongs. This is especially true since eloquence, as an attribute of a comment, will soon be determinable even by machines. Generally, a lengthy comment would almost always have to be voted up because of the effort put into it, so it might as well be given a higher starting value determined by automatically counting its words in such a scenario.
Reality is different anyway.
Most people have long realized that voting up comments according to their presentation makes no sense whatsoever. That's why most people rate comments up based on whether they agree with them. But because of the consequences (complaints, perhaps even mails to the editors of the site), few people have the guts to vote comments down which are nonsense.
The result is that the rating system is used not as a mechanism to vote on comments, because of the resulting emotional frustration, but rather as a mechanism to select the comments which are considered good. But as was pointed out before, the rating system isn't built for this kind of selection. Scores are averaged on a scale from 1-5. Now, if one group of users selects one half of the comments and rates them as 4 or 5, and another groups of users selects another half of the comments and rates them as 4 or 5 (the part they agree with), the result is that every comment is somewhere in the 4-5 range, perhaps with the shorter comments in the 3-4 range. Incidentally, this is the situation on K5 right now (with individuals instead of groups, of course), which makes the rating system mostly worthless, were it not for the few users who dare to use the ratings on the lower half of the scale.
The proper rating discipline
If you want to rate in a way that improves the discussion quality on the site, here's my advice:
- If a comment only contains opinion, and no statements of fact (e.g. "The kid should be electrocuted!"), rate the comment according to your opinion (value system) and knowledge on the subject.
- If a comment contains statement of fact, but you only have an opinion (i.e. emotions resulting from your value system) on the subject, but no real knowledge, do not rate the comment.
- If a comment makes a point of which you are aware, which is true to your knowledge, but you deem it as little-known enough to warrant further exposure, rate the comment up. For the difference between and 4 and 5, you can use such factors as presentation, emotional agreement and minor errors as guidance.
- If a comment informs you about something you didn't know previously, and you see it as sufficiently backed up by solid facts, rate the comment up.
- If a comment makes a point which makes you think, but you do not know whether it is true or not, do not rate the comment.
- If a comment makes a point which makes you feel bad, but you would not be able to explain why if the author inquired why you rated his comment a 1 or a 2, do not rate the comment.
- If a comment contains false logic, lies or errors or otherwise undesirable content, and you could elaborate further on that if that became necessary (but do not have the time nor the will to do so), rate the comment down.
Generally, low ratings are frustrating and discouraging. That's the real and only reason people complain about them, the seemingly rational reasons they give for this behavior are, as we have found, illogical and inconsistent. Consider their emotional impact when you give low ratings and use them especially where you think the subject is of such importance that lies and disinformation should not be spread.
Of course, replying is always most desirable, but there are cases when the point in question has already been made in several previous comments and the comment in question has simply ignored them, or when you don't have the time to write a detailed reply (the most common reason to rate instead of replying).
In the long term, the rating system must evolve in order to reduce the amount of noise that it will inevitably continue to produce and in order to increase its usefulness. Reducing the emotional frustration caused by low comment rating by building a system that aggregates scores should be a primary goal - this would preserve the usefulness of the ratnig system, as comments with low total scores would be equivalent to those with low ratings, but not as frustrating. Eventually, a more personalizable system of user groups and trust would be preferable.