Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

[P]
Making the Perfect Moderation System

By nebby in Internet
Sun Oct 01, 2000 at 02:58:14 AM EST
Tags: Internet (all tags)
Internet

With all the recent posts concerning ideas behind moderation and content control for forum sites like this, I thought I'd give a detailed rundown of the new system I've been creating called Glasscode. It's a new software along the lines of Slashcode and Scoop, written using Java servlets, that has quite a complex moderation/point system that's easy for users to use and will (hopefully) be effective.

Hit "Read More" for a mathematical rundown of how it works, or jump right to half-empty, my debut site running the system. I'd love to get feedback to tweak the system or even flat out explanations as to why it probably will or won't work :)


(I know I posted the Glasscode/Half-empty story already, but some people wanted a detailed description of the rating system, so here goes :))

Glasscode is a non-submission queue system. That is, everything people post gets put on the site, immediately. To keep things sane, however, there is a point system involved which will restrict when, where, and how often users can post depending on their performance.

The posts to the site are content rich blobs called "Ideas" .. much like kuro5hin submissions but with the addition of file attachments. These Ideas appear much like stories on kuro5hin or /., complete with comment threads. The main difference in the appearance of both Ideas and comments, however, is the presence of three small "Rating" buttons. A green plus, a red minus, and a yellow neutral one. As readers peruse the site, they can rate all comments and Ideas with a single click (1-Click Rating? :)) b/c of a window which spawns in the background.

Every user has a certain amount of points associated with their account. Ideas and Comments also have point values, which is the sum of all the pluses and minuses people rate them with. (Neutrals don't do anything but increment the rating count for that post, more on this later.) The math involved is such that a user's points provide both a "karma" level and also reflect the users ability to post. The way this is taken care of is as follows.

Every Idea on half-empty (or any Glasscode site) gets posted into a category. There are the root categories (Art, Computers, etc.) which are divided into subcategories. (Currently, users are not allowed to post into root categories, but this is an option.) Each category has two numerical values associated with it. The "point percentage" is a percent value which represents the percentage of a users points they "risk" when posting into the category. For example, if user Joe with 100 posts an Idea into the "Cars" category, with a point percentage value of 25%, Joe can gain or lose up to 25 points on his Idea. Also noteworthy is that Joe cannot risk more points than he currently has between all of his Ideas and Comments. The second value associated with a category is the "minimum point cost" .. this keeps folks with low points from posting. If Joe's 25% "risk" didn't come up higher than the minimum point cost, then Joe cannot post into the category.

How are these numbers determined? Well, every five minutes Glasscode does some statistical work to set them. First, it calculates the point mean of all the users, and the standard deviation. It then counts the number of Ideas running "Live" on the site (more on what this means later) and calculates the percentage of the total Ideas in each category. Using these percentages, it assigns a point percentage cost to each category between a manually set min and max value. The most active categories get closest to the max, the inactive ones get the min. As for the minimum point cost, it again uses at the percentage of total Ideas in each category. If 10% of the Ideas are in category A, the minimum point value is set so that the top 90% of users will be able to post into it. This is calculated using the normal curve generated from the point mean/SD. Pretty odd to actually be using statistics I learned in high school, but it seems to work in a fair manner.

Well, as stated earlier, every time you post something, you "risk" a certain amount of points based upon the percentage cost associated with the category. If you have 100 points, for example, you cannot "risk" more than 100 points across all of your postings until you get a point reward.

So the question then becomes, how does one increase their points? The key is the rating system outlined above. When a user posts an Idea (or a Comment within an Idea) .. the Idea is "Live" for certain number of days. After the time is up, the Idea is archived (along with its comments) and points are added or deducted to each user depending on their posting's performance. This performance is measured in a few ways. First, the rating system determines the ratio of the points earned to the number of votes (kind of like kuro5hin) .. so now we see the significance of the Neutral button. A little bit extra is added onto this ratio for every reply to the Idea/Comment, and every time the Idea was read by a unique user. This final number, between 0 and 1, determines how much of the "risk" taken is given back to the user or taken from the user. There is a manually set variable to tell Glasscode what ratio value to take as making the user break even and not gain or lose points. Getting complex now, eh? :)

One more thing to note is that small amounts of points are awarded for rating stuff. This is to make sure nobody digs a hole they can't (albeit slowly) climb out of.

Of course, all this is magic running in the background. The users rate and get a magical "Post" button when reading in categories that they can post into at that given moment. Their points go up and down each day as their posts close out. The point value for users, when calculated in this way serves a few different things:

1) A "karma" rating. Users with high points have posted popular, consecutive Ideas, because their "risks" are percentages. In order to get to say, 300 points, you couldn't do it in one jump. You'd have to make a few good posts to get your percentages to a point that you could reach 300.

2) Posting ability. Users with high points can post in the most active categories (because of the normal curve) and will be able to post more because their "risks" will bottom out to below the minimum point cost for each category in more steps.

3) Bonus points. Users who are in a high enough percentile on the normal curve get bonus points tacked onto their Ideas/Comments when posting (like the +1 bonus on /.) These aren't included in the rewarding process, but cause their Ideas to show up on the front page more often and have their Ideas/Comments appear higher in the listings if the user is sorting by highest.

The front page is generated each hit, drawing Ideas out from the pool based upon their youth & popularity.

Whew. That's alot of stuff :) Hit me with some constructive critisism! One thing specifically I've been stuck on is if there is a better way to make sure higher point users have even more posting ability, but still have to take the associated "risks." The current system gives a user with a lot of points a few extra posts between archivals, but not that many, because the "risk" is based on a percentage. Thanks! :)

Sponsors

Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure

Login

Poll
Will this system work?
o Yes 45%
o No 34%
o Glasscode sucks 20%

Votes: 79
Results | Other Polls

Related Links
o Scoop
o Kuro5hin
o half-empty
o Glasscode/ Half-empty story
o Also by nebby


Display: Sort:
Making the Perfect Moderation System | 49 comments (44 topical, 5 editorial, 0 hidden)
Wow. Big Idea... (3.40 / 5) (#6)
by magney on Sun Oct 01, 2000 at 04:26:42 AM EST

so big there's no room for Comments! :-)

Seriously though, I'm curious about half-empty now. I think I might go play around with it a bit. Really, it's hard to say ahead of time whether an idea like this will work unless it's been tried. On another note, I'm amused by how the negative vote has been split between No and GlassCode Sucks - if I were a cynical sort, I'd suspect that "GlassCode Sucks" was put in to split the negative vote. :-)

Do I look like I speak for my employer?

Re: Wow. Big Idea... (3.00 / 3) (#7)
by nebby on Sun Oct 01, 2000 at 04:42:19 AM EST

Yeah the poll seems pretty even :)

Glad you like the concept. I really need to find a way to get the word out though.. coding seems to have been the easy part, marketing the harder part. Ugh.
Half-Empty: A global community of thoughts ideas and knowledge.
[ Parent ]
The best moderation system. (3.50 / 10) (#8)
by Paul Dunne on Sun Oct 01, 2000 at 05:28:53 AM EST

Here's the code for the best moderation system:








(For those of you who just got out of bed or something: yes, that's right, none.)
http://dunne.home.dhs.org/

Re: The best moderation system. (4.33 / 3) (#13)
by squigly on Sun Oct 01, 2000 at 07:05:51 AM EST

I'm not sure. Some editorial input is allowed. Just for the "Hot Grits, Natalie Portman, my country is better than yours, k5 sucks etc...." type posts.

The story moderation on K5 is quite useful, although it does need some improvements which have been discussed to death. Comments I feel are either on topic or rubbish. Outside of that, whether its a "good" comment is highly subjective.

--
People who sig other people have nothing intelligent to say for themselves - anonimouse
[ Parent ]
Re: The best moderation system. (4.33 / 3) (#17)
by Paul Dunne on Sun Oct 01, 2000 at 10:06:35 AM EST

Yes, some editorial control is essential. Hot Grits etc just get deleted. But a scoring system for posts, which is what moderation is, is a cure worse than the disease. Everyone should have an equal voice; it's up to each reader to decide for themselves the worth of a post. Moderation systems effectively make that choice for the reader, by ensuring that low-rated posts will either a) not be seen at all (as on /.), or b) be sorted "out of view", as can be done on Kuro5hin. The Kuro5hin system as it stands is not too bad, but only because, short of sorting by rating and only reading the highest-rated comments, one can't avoid low-rated comments. The big problem on Kuro5hin with moderation is that of course people will use it to express their opinions. Someone who disagrees with a post should have the courage of their convictions, and post a reply -- always assuming they have something to say. Giving them an anonymous outlet for their spleen merely attracts an undesirable type of user.
http://dunne.home.dhs.org/
[ Parent ]
Re: The best moderation system. (4.00 / 1) (#26)
by Sunir on Sun Oct 01, 2000 at 02:44:09 PM EST

As I stated above, this is what a wiki does. Actually, the moderation system is entirely based on peer review, but it is not overt. Instead, if material like "hot grits" gets posted, your peers will edit the page and remove it. There is no editorial control beyond what people do with their hands.

If another member notices through RecentChanges the "editorial" decision, she can do several things.

  • Let it be (good decision, not interested, unsure what to do)
  • Reverse it
  • Leave a note to the original author and the editor.
  • Reverse it and leave a note asking for clarification.
  • Edit the text so it reflects any viewpoint in between the two authors, or even a new one. Or whatever.

The point is, there systemic restrictions aren't as constrained as they are on a web log.

"Look! You're free! Go, and be free!" and everyone hated it for that. --r
[ Parent ]

Re: The best moderation system. (5.00 / 1) (#28)
by Rainy on Sun Oct 01, 2000 at 08:47:23 PM EST

I'd agree with you BUT only when the total amount of posts can be easily read in one sitting - which is only true for really tiny sites. Take usenet, for instance - nobody reads all of it, it's impossible. I come to slashdot and kuro5hin instead of usenet because I can read 10 posts on each story and know that even though I haven't seen some really good posts that weren't modded up for whatever reason, the ones I did read are quite a bit better than average. That's the whole point, isn't it?
--
Rainy "Collect all zero" Day
[ Parent ]
Re: The best moderation system. (5.00 / 1) (#31)
by Paul Dunne on Mon Oct 02, 2000 at 02:46:46 AM EST

That is that point. Except that what you describe is not what happens. Case in point. If this were slashdot, I had now lost karma (oh no!!). For what? Trying to shut up an obvious know-nothing. What has happened to /. is starting to happen here: hordes of clueless, Windows-using kids with too much time on their hands are deigning to share their vast wealth of knowledge and experience with us, both by posting and (worse) by moderating. If we want to see this *not* to happen, we definitely can't rely on any moderating system.
http://dunne.home.dhs.org/
[ Parent ]
Re: The best moderation system. (none / 0) (#36)
by Rainy on Mon Oct 02, 2000 at 10:48:14 AM EST

Have you read my post carefully? That's exactly what I said - some good posts do get lost in the moderation debris, but the whole point is that the ones on top are of much higher quality than average. I agree with your post though - linux is a good desktop system for people like me, who mostly need email/web browsing. I'm not too sure about people who need a good office suit and good finance software. It's coming, though.
--
Rainy "Collect all zero" Day
[ Parent ]
Re: The best moderation system. (none / 0) (#38)
by Paul Dunne on Mon Oct 02, 2000 at 10:55:11 AM EST

Yes, I did read your post. I just don't agree with it! Your conclusion that "the ones on top are of much higher quality than average." seems to me unsupported by the evidence. Indeed, I'd go further and suggest that the reverse is often the case: rubbish is moderated up by those who, like the poster of the rubbish, simply don't know what they are talking about. If these are in the majority, then a moderation sytem not only doesn't help, it makes matters worse; if these are in a minority, then there's no need for moderation in the first place.
http://dunne.home.dhs.org/
[ Parent ]
Re: The best moderation system. (none / 0) (#47)
by Rainy on Tue Oct 03, 2000 at 11:38:50 AM EST

In that case, why have /. and kuro5hin? Usenet, according to you, is far better. Why are you using this site? Usenet is exactly what you want - news with no moderation. Seeing however that you *do* use kuro5hin, let me suggest that perhaps you do find moderation useful but you are annoyed at some people moderating unwisely, which is unavoidable. It's like democracy: it's as good as average IQ of citizens, but even IQ is fairly low (alas, it is), it's still better than a moncarchy or tyranny.
--
Rainy "Collect all zero" Day
[ Parent ]
Re: The best moderation system. (none / 0) (#48)
by Paul Dunne on Tue Oct 03, 2000 at 01:15:31 PM EST

Kuro5hin is not (yet) moderated in the strict (/.) sense of the word. There is a posting scoring system in place, which is kindof like moderation, but with the significant difference that no post can be rendered invisible by a setting in your user preferences. Kuro5hin is *edited*, a crucial difference. Hot Grits et al aren't "mod'ed down"; they are deleted. The comparision to Usenet I fail to understand. I think you're simply misuing the term "moderated". Kuro5hin is like a moderated mailing list or newsgroup in the sense that crap will be deleted. But "moderation" in weblog terms doesn't mean that, it means posts being scored, and in the case of /. rendered effectively invisible, by user intervention. For the third time: this doesn't work. What happens with /. moderation is that Q. Random Reader decides what you'll see and won't see. On Kuro5hin, the influx of clueless Windozers may not do permanent damage the way it did to /.; but the danger is already evident in the way posts are being scored. And it's not just the most clueless people; everyone has a tendency to "mod down" stuff they don't agree with, think is a flame, etc, etc. I have a basic disagreement with this. All posts are created equal, and should stay that way unless they are shit, in which case they should be removed.
http://dunne.home.dhs.org/
[ Parent ]
Re: The best moderation system. (none / 0) (#49)
by Rainy on Wed Oct 04, 2000 at 03:25:10 PM EST

Kuro5hin is not (yet) moderated in the strict (/.) sense of the word. There is a posting scoring system in place, which is kindof like moderation, but with the significant difference that no post can be rendered invisible by a setting in your user preferences. Kuro5hin is

Um.. that's purely academic difference, it still boils down to whether a particular user wants to see all posts or higher scoring. On /. he may set threshold, here he'll simply read few higher scoring posts and then stop (like I do when I have little time or when topic isn't very interesting to me).

Now you say that you don't understand the reference to Usenet. What I meant was that the only advantage of ./ and kuro I see over moderated newsgroups is that I have a choice of reading just a few comments that are considered the most valuable by most readers. In usenet, you can't do that, instead you do exactly what you say you want: read all posts and decide for yourself whether they are worthy or not. Usenet even has an advantage: it's sortware was specifically designed for that purpose while browsers weren't. So, in slrn you can hit j to read the next post, and handle it as a separate unit, for instance saving it to a file. If you want to save a weblog post to a file you'd have to copy/paste it manually. That's just an example, newsreaders have many many other things that make newsreading easier and faster. To reiterate, the only thing I personally see going for weblogs is the moderation that helps me save my time.

What happens with /. moderation is that Q. Random Reader decides what you'll see and won't see. On Kuro5hin, the influx of clueless Windozers may not do permanent damage the way it did to /.; but the danger is already evident in the way posts are being scored.

Now this I totally don't understand.. *anyone* can go and set threshold to -1 at /. if he doesn't like to be at the mercy of detestable Q. Random Reader. It's up to every individual. That's what's freedom about: having a choice. Again, if you think having this choice is bad, usenet seems to be perfect for your tastes.. Tell me something: what's wrong with usenet? To me, your position seems to be something like this 'I want weblogs to be exactly like usenet'. But we already *have* usenet! Hey, did you also notice that many sites have weblogs with no moderation, like zdnet and salon for instance, and how unpopular they are? Guess why.

And it's not just the most clueless people; everyone has a tendency to "mod down" stuff they don't agree with, think is a flame, etc, etc. I have a basic disagreement with this. All posts are created equal, and should stay that way unless they are shit, in which case they should be removed.

Sure, I mod down the stuff I disagree with, that's the whole point. I disagree that it's *true*, so how can I leave the post that I think is a lie to be on top? I may be subjective, in fact I can't help being subjective, but so is everyone else. There are often a few posts that just give facts on the story, or factual corrections, and they always get modded up, cause nobody thinks they're untrue. How do you tell an opinion from a flame though? It's highly subjective. Okay, sometimes it's a plain flame, but mostly it's a mixture and then it's a sliding scale. Secondly, all posts are not created equal. They're all created different, unless they're double postings (like when you hit submit 2 times). I know you're referring to often quoted 'all men are created equal', and misusing it horribly. The point of that quote is that all people should be given equal chances to succeed: nobody should be forbidden to study art merely because he was born in farmer's family. It doesn't mean that every farmer who knows no art whatsoever could go to Louvre and post his painting there. It doesn't mean that any bum can approach Gates and say 'hey, give me half your money, we're s'posed to be equal'. Well, upon reflection, if that bum is me, I don't mind all that much :-). Anyway, that's the point - all posts are created equal in the sense that they all start with (none) score, but after that they're judged by their merits and scored. Again, if that seems a bad idea, look into usenet: it doens't have that. :-) Don't take it as mean, I'm not saying go away from kuro if you don't like it, I'm merely saying that from what you read, you seem to be looking for a system exactly like the one Usenet uses.
--
Rainy "Collect all zero" Day
[ Parent ]

Philosophy of moderation. (4.90 / 10) (#9)
by inspire on Sun Oct 01, 2000 at 06:18:40 AM EST

Spending more time worrying about what users submit rather than producing/serving content seems counterproductive. It seems that the three big (IMO) weblog players are all proud of their respective "behind the scenes" work -

  • Slashdot - arguably the first of its genre and with an interesting moderation, metamoderation, karma system
  • kuro5hin - public story queues, mojo
  • Advogato - weird trust-web metrics

However, I don't visit these sites to do my "moderatorial duty". I visit these sites because I am interested in the quality of discussion and articles each has to offer. Whether a system implements mojo, karma, web-of-trust, or whatever, it all boils down to two factors (and I'm sure I am preaching to the choir here)

  • signal - the good stuff - discussion worthy, well argued comments/articles
  • noise - crap from trolls, newbies who dont know the rules, spam, etc

The ultimate goal of any moderation system is to /maximise the signal:noise ratio/, that is, provide users who want to speak with an adequate opportunity to be heard. This is where I think the Glasscode system falls down - although I haven't used it (and correct me if I'm wrong), the system proposed makes it more difficult for users to post to more active areas of the site, "balancing" the content.

This is not the goal of moderation. Having basically an "elite" area where only high karma users can post seems to be exclusionary and ultimately self-defeating. If users want to discuss something, making an area of the site "active", what is the logic in then restricting that area to high karma users to post only?
--
What is the helix?

Re: Philosophy of moderation. (4.66 / 3) (#10)
by Inoshiro on Sun Oct 01, 2000 at 06:29:52 AM EST

Why should we (the admins) be be proud of our behind the scenes work? Whole we do provide the infrastructure and monitor things, it's the users who should be proud -- they set the standards and the content for this site. We just try to live up to it :)

--
[ イノシロ ]
[ Parent ]
Re: Philosophy of moderation. (4.00 / 3) (#11)
by inspire on Sun Oct 01, 2000 at 06:35:06 AM EST

Why should we (the admins) be be proud of our behind the scenes work? Whole we do provide the infrastructure and monitor things, it's the users who should be proud -- they set the standards and the content for this site. We just try to live up to it :)

Ahem. I didn't mention the admins being proud of the behind the scenes work - I just said "the three big players are proud of their moderation systems" (okay, somewhat paraphrased).

Perhaps your Freudian slip is showing? ;-)
--
What is the helix?
[ Parent ]

Re: Philosophy of moderation. (3.66 / 3) (#12)
by inspire on Sun Oct 01, 2000 at 06:40:12 AM EST

All that said, perhaps the best moderation system is to just choose your users wisely :-), a la Advogato. I doubt that will score many points with people as the site's whizzy new userfilter system spits out:

"Unfortunately, our automated search on Deja shows that you are a clueless 5|<r1p7 |<1dd13. Get lost".
--
What is the helix?
[ Parent ]

Re: Philosophy of moderation. (5.00 / 3) (#14)
by rongen on Sun Oct 01, 2000 at 07:27:34 AM EST

noise - crap from trolls, newbies who dont know the rules, spam, etc

Is this the only source of noise (posting "dumb" stuff)? As these forums grow larger and larger (and they will, as the web grows) there will just be far too much content for an individual user to meaningfully choose what to read easily. It's like when you use a search engine you know that you are missing 10 pages for every page you get (or whatever). We really need an algorithmic auto-moderation system that can actually find content you will like to read (and keep an eye on ongoing discussions, letting you know when something interesting pops up). This would in no way prevent you from browsing "at -1" but it would make the quality of the discussions you read much higher on a personal level and allow discussions to go on for long periods of time which would alleviate the need for people to post things before they have had time to fully develop thier arguments.

So my point is that noise is basically anything I don't want to read. This doesn't mean opinions I am not interested in or disagree with, it means any subject I have already seen posted and discussed on some other weblog, or maybe a topic that gets covered a lot that I think is dull, or whatever.

The idea that manual moderation will somehow be able to keep the noise down is really a temporary one. It's a band-aid for the problem. Slashdot did it (among others) and many sites followed suit. Now it seems like everyone (but Technocrat) is moderating. This is an indication that soon this will not be enough. People won't be able to keep up, and too much noise will get modded up (by jerks) and too much signal will stay on the floor (through apathy, or whatever).

One of the best things about the web is that the first time I see a site like "perpetual bubble wrap" or "hamster dance" (two deliberatly bad examples!) it feels like I discovered them. When I read a web page for the first time it is also like this (the same argument goes for forums). Now, let's say I am really interested in a discussion but only locate it months from when it was started (e.g. as a result of a Google search). I craft a very nice response that is intelligent and might even help solve the problem (I need you to use your imagination here!). What are the odds that this will get read by anyone? Very, very low. Unless you have an automatic mechanism for evaluating the contribution and notifying users that this contribution is extremly relevant to them. Those interested readers might check it out after all.
read/write http://www.prosebush.com
[ Parent ]

Re: Philosophy of moderation. (none / 0) (#34)
by interiot on Mon Oct 02, 2000 at 08:53:49 AM EST

I was with you, up until the third paragraph where you started promising me the world.

For me, noise is a post that isn't ontopic WRT its parent or its story, a poster who didn't fully understand its parent or story before posting, a post with a lot of logical falacies, or a post that presents opinion as fact without backing it up.

If such a program were written, I'd call it human. I personally don't think that the state of the art is up to the task yet. I haven't seen any AI program work in the real word that tries to do much beyond lexical analysis. Heck, syntactic analysis wouldn't even be enough here.

Granted, AI could help automatically moderate posts to some extent (Troll Terminator 2000, basic lexical analysis). But when you expect AI to be able to do everything for you (saying that manual moderation is a temporary band-aid), I think you're just a tad optimistic, at least for the next couple years.

[ Parent ]

Re: Philosophy of moderation. (none / 0) (#35)
by rongen on Mon Oct 02, 2000 at 10:03:49 AM EST

OK, I agree that what I am talking about is not totally possible right now (especially with the hardware and aged compiler, etc, we are using on our site) but the basic concept can be used to auto-moderate.

Clustering algorithms are expensive but can work. Remember the goal here isn't to say "you will ONLY read this subset of the posts" it is to guide people by suggesting things. Current moderation basically does this by imposing a mask over some articles. On slashdot, for example, I find that many articles that are on-topic but arrived "late" are not moderated up because the moderators don't want to wade through 250 -1 ranked posts to find the diamond in the rough, and who can blame them?

imagine a system that pays attention to things you have found interesting in the past and classifies new documents based on the similarity with past stuff, and also uses the degree to which others read this document to weight it. This is all I am talking about. You like Hockey? You will read lots of stuff about it in the sports news and possibly ignore synchronized swimming, for example, so posts with "hockey", "referee", "puck" are more interesting to you than posts about synch. swim (the words for it are not even in my vocabulary which actually illustrates my point, I think).

Does this still sound too good to be true?
read/write http://www.prosebush.com
[ Parent ]

Re: Philosophy of moderation. (4.33 / 3) (#16)
by puppet10 on Sun Oct 01, 2000 at 09:54:45 AM EST

I believe the ability to post Ideas (ie articles rather than comments although it does take some points to post comments) is limited by the active not active areas of the site. It seems this is a generally good idea to self limit the number of articles posted in a category per time period. Thus if many articles were going only to the computer section and filling it up it becomes more costly to post an Idea and the risk goes up. Thus at some point only longer standing users can post new ideas to a heavily used area and they had better be good (which if it is a long time user they probably will have thought carefully about posting) because they stand to lose more points if it is deemed bad. However in less heavily trafficed areas newer users can post and generate disscussion in areas which haven't been explored as fully.

So the disscussion isn't exclusive, but putting yet another article in the computer/copyright queue for people to look at and discuss is potentially more costly, and you might need to invest in the site a bit in order to post it.

The moderation system for Glasscode seems to deal with a number of things; reducing the number of article posts in a specific area or by one user to a reasonable amount, reducing the ability to troll/grit pour/spam articles especially in heavily posted/read areas of the site, and giving people who contribute to the site in a positive way more possibility to do so again in the future.

That said the system isn't perfect and there probably isn't a perfect system to allow free flow of ideas and comments without opening the disscussion up to the few who seem to want to ruin it (eg. Slashdot trolls/hacking/spamming, Kuro5hin spamming/crashing, and Usenet spamming) but it is another attempt and is oriented in a slightly different way to promote a different type of community than on /. or Kuro5hin.

[ Parent ]
moderation = limited censorship (2.80 / 10) (#15)
by xah on Sun Oct 01, 2000 at 07:55:08 AM EST

Look, it's pretty simple. If you prevent people from voicing their system, it's a form of censorship. What this "glasscode" purports to do is just that. It limits who can post. Certain people's voices are inherently valued more than others. It's just not fair.

Even a fool says something insightful on occasion. It is wise to keep one's ears open for such moments.

Limited time = need to choose (4.50 / 2) (#19)
by Rocket surgeon on Sun Oct 01, 2000 at 12:20:21 PM EST

It's a matter of time available. Most people have to allocate their time. Most people don't want to read everything in order to find the nuggets.

Do you watch every single TV program?

Do you read every article in the encyclopedia?

Do you read every book in the library?

[ Parent ]
Re: moderation = limited censorship (4.00 / 2) (#20)
by Pakaran on Sun Oct 01, 2000 at 12:22:15 PM EST

Saying it limits who can post is rather ridiculous. Log on there; you can post to anything. I could when I first logged on, before I got any points. And, after a week or two, you can post as many articles as any sane person would want to.

As to your other remark, how can a system deny you freedom of speech when it makes no distinction based on the content, or length, of what you have to say? If I wanted to, I could log on right now and post an idea with a 5 meg .tgz file of every anarchy 'phile' in textfiles.com! To be sure, I'd probably lose about 30 points, but that'd be after the fact. And it means no more than it does to say that my friends would lose respect for me if I started ranting to them 24/7 about that sort of thing.

[ Parent ]

Re: moderation = limited censorship (4.00 / 1) (#23)
by nebby on Sun Oct 01, 2000 at 02:17:46 PM EST

The point system isn't meant to censor stuff. It's meant to keep people from spamming, and it's meant to give people some pride if they post good consecutive ideas.

In regards to other comments ..I have it setup to hide comments if you haven't rated an idea because it allows more non-biased rating. Since this seems to draw away from the quality of the site, I guess I'll be turning that option off :)

Half-Empty: A global community of thoughts ideas and knowledge.
[ Parent ]
moderation technique (3.16 / 6) (#18)
by base_16 on Sun Oct 01, 2000 at 11:51:21 AM EST

There is also something much like this used at perlmonks. Their rating system is a little simpler, but still works much along the same lines. This system uses experience points and depending on posts, etc. you get a higher rank.
25% chance of +1 every time you vote
+1/6 number of votes cast if all votes are cast on any given day (This bonus only exists until you reach level 5)
25% of gaining 2 XP points once a day if you were logged in within the past 24 hours
Here's how the votes on a post affect things:
norm- This variable is calculated daily when votes are calculated. It is the average reputation of all of the nodes created within the past week.
rep- The current reputation of the node being voted on
Range - Odds of a Gain - Odds of a loss
$REP<$NORM - 1/3 - 1/3
$NORM<=$REP<2*$NORM - 1/2 - 1/3
2*$NORM<=$REP<3*$NORM - 2/3 - 1/3
3*$NORM<=$REP<4*$NORM - 3/4 - 1/4
4*$NORM<=$REP - 1 - 0

Plus, the more points you get, the more times you are allowed to vote. Anyway, I just thought I'd mention how someone else does things... this system seems to work and it's not quite as complex as the one in Glasscode.
--
this signature will be affixed to each of your comments
I like Everything too. (3.00 / 1) (#25)
by pin0cchio on Sun Oct 01, 2000 at 02:35:02 PM EST

Perl Monks uses the Everything engine from Everything Development, a division of Blockstackers (which sold TOS to the company that would later give K5 a server). Everything seems to be primarily designed for building a huge database, not for creating discussions; the Everything site I frequent (Everything2) doesn't support threaded/nested discussion.
lj65
[ Parent ]
Moderation getting in the way of conversation? (3.71 / 7) (#21)
by rainbowfyre on Sun Oct 01, 2000 at 01:05:32 PM EST

I was interested in seeing how the system worked, so I hopped on over to half-empty. There were some interesting Ideas, and I was interested in learning more about particular topice. I couldn't for the life of me figure out how to read any comments! There was just a little tag on the bottom that said the comments would be invisible until the Idea was rated, and urged me to sign in and begin rating.
I am afraid I don't understand. Why would you want to hide a good conversation? I wasn't interested in posting, just hearing other people thoughts. Do I have to sign in to do that?

-rainbowfyre
Vericon is coming!
Re: Moderation getting in the way of conversation? (4.00 / 1) (#27)
by nebby on Sun Oct 01, 2000 at 02:53:12 PM EST

I turned off this option now, it seems that most people don't like it very much. The basic idea was that if you couldn't read comments until you rated an idea, you would make a more un-biased rate.

I guess the tradeoff isn't worth it if i'm losing users though :)

Half-Empty: A global community of thoughts ideas and knowledge.
[ Parent ]
Two things (3.66 / 9) (#22)
by yuri on Sun Oct 01, 2000 at 01:06:04 PM EST

Your site wouldn't let me view comments until I rated the idea. I couldn't rate the idea until I had logged in. You will severly hamper your site if you make it this difficult to sample the site. I'm not going to create an account just to see if the site is worth my attention. You just lost one new member.

Generating your front page for every view is a nice idea for a small site with few hits but will have to change if you ever get popular (see first point).

Cheers



Wikis (3.66 / 3) (#24)
by Sunir on Sun Oct 01, 2000 at 02:33:51 PM EST

I'm not really convinced moderation, certification or story ranking is absolutely necessary. The wiki model is stable and lacking in these hard security measures. Instead, they rely on good, ol' fashion soft security and other community solutions.

For MeatballWiki, instead of having only one ranking system, we have written several indexing schemes. Notice how most of those indices are on a third-party site from the wiki host.

By the way, for those unaware of what a wiki is, think of the web but with write permissions. There's a lot more to it, like a special WikiSyntax and the ubiquotous LinkPattern [sic]. If you would like to know more, check out MeatballWiki where meta-issues are the flavour.

"Look! You're free! Go, and be free!" and everyone hated it for that. --r

Algorithm problems...? (4.40 / 5) (#29)
by aralin on Mon Oct 02, 2000 at 12:44:59 AM EST

Well, seems to me that there are few problems with this algorithm. One thing that I can see in first look is that something is wrong with counting all the statistics each 5 minutes. That even sounds wrong. They won't obviously change so much and especially not for all users. With good data structures design you could just make only incremental changes. And recount all statistics let's say once a day to be sure that you don't have any mistake in incremental algorithms.

Another thing is that you can recount statistics for different users only at the time that its expected that there will be some change. His story gets a comment, he send a comment, his idea get archived, depending on how you exactly want to rate it all. Also with well desgined data structures and some of the values being computed dynamically only when they are actually used, you would not need to recompute most of values, just change some of attributes on fly.

And definitely there is no point in recomputing values for users that are not active. Since they won't use them. So lets say that you can just skip some user until he will get actually logged. I am kind of sure that the server will compute everything, before he will get reply with the front page... :)

In case you would be more technical, use a bit more formulas instead of zillions of words, I could try to find some problems with the actual algorithm proposed, but this way it seems too much work...

Re: Algorithm problems...? (4.00 / 1) (#30)
by micboh on Mon Oct 02, 2000 at 01:20:33 AM EST

I'm going to GPL the source eventually, so you'll have the chance to see what's going on :)

As for the statistics, they're calculated in about 1 or 2 seconds right now so the 5 minute spawning will work for now. How fast it will be once things get rolling.. well I dunno :)

The data structures don't really require all the checks I do during the statistical work in alot of ways every 5 minutes, but it's a good sanity check on things while the site is still going to probably have bugs in it.

-Greg

[ Parent ]
self selection (3.66 / 3) (#32)
by mrr on Mon Oct 02, 2000 at 08:42:46 AM EST

I believe your system works best to enable a group to reach a consensus point of view. Over time the the voting and points will reinforce views and values shared by the community using the system. Voices that best express those views and values will emerge as The leading speakers. Contrary points of view will slowly be stifled.

Unlike another poster, I don't feel that this is censorship. The community voting is providing feedback to posters and negative feedback translates, eventually, into reduced posting rights. This seems like a fair way to have a community define itself through its writings.

However I believe that providing a platform for dissident voices can be important. /. does this with moderation and the ability of readers to set their cutoff level. The noise that is posted is available to anyone who wants to read the stuff that's been moderated down.

An interesting variation on your system would be to change the effects of the voting so that a moderation level is assigned to posts as a result of voting. Rather than affecting the ability of someone to post, it would control the visibility of their postings. This would keep dissident or unpopular voices availabe, and perhaps even generate subgroups on your site. Which leads me to think about the feasibility of not making the ratings global, but have more than one area on a site with independently maintained ratings. Someone rated high in area A might fare poorly in area B. Or perhaps low rated comment threads could be voted out into spawning their own topic area.

You system sounds good, and it's stimulating some thinking on my end.

An idea (2.75 / 4) (#33)
by codemonkey_uk on Mon Oct 02, 2000 at 08:50:06 AM EST

Some thoughts:

- Forum.
- Moderation.
- Portal.
- Customisation.
- Relativity.
- Trust.
- Automation.
- Community.
- Scale.

It might be too much for current servers to implement on any scale but I have been feeling recently that the personalied customisation of portal sights can be applied on a grand scale to the moderation of disucssion forums. Communitys can build themselves. My moderation can be used as a filter for what I see as moderated by others, that is, when I load the page the moderated, filtered forum I see is moderated/filtered by the users I have moderated up or moderated in agreement with in the past. Special interest groups can thrive without without conflict. Perhaps. A forum need not be a single united comminity, but multiple, interacting communities!

Just a thought...

Thad
---
Thad
"The most savage controversies are those about matters as to which there is no good evidence either way." - Bertrand Russell
a political example (4.00 / 3) (#37)
by Jim Madison on Mon Oct 02, 2000 at 10:52:07 AM EST

I'm not sure what the "perfect" moderation would be, but we've created one for a political community that is based on civic values.

What does that mean? To us it's "equality of opportunity to participate." Technically it is similar to the glasscode system, there are "articles" (instead of ideas) and comments. We don't have article queuing, but every article is guaranteed to be rated by at least ten users. Ratings decay exponentially over time to ensure that the top articles are timely and relevant.

In addition to the technology, we have other for making our site a meaningful place for democracy. We also spend a lot of time trying to get policy makers to participate (through an online debates , interviews, etc.) and have spent some time considering privacy policy, rules of engagement, etc., which are an important part of the broader set of issues for creating a vibrant online community.

http://www.quorum.org

My partner is more eloquent than I am, and you can read his thoughts in this interview of Scott Reents on /.

Got democracy? Try e-thePeople.org.

Democracy sucks (4.00 / 3) (#39)
by blamario on Mon Oct 02, 2000 at 04:41:05 PM EST

It seems to me that any kind of one-moderation-for-all can only work on small scale. I mean, look at the /. S/N ratio. When so many people with so many different opinions get together, you can't avoid flames. And you can't expect any kind of moderation to fit everybody. In the best case it might fit majority. That's when the minority starts trolling.

The obvious solution is for minority to create another forum where they can discuss another point of view. But it would be much better if the comments could be the same across all these separate and only the moderation different. For example, one day you decide you're in mood for some M$-bashing. You go to forum, choose the bashing option, and you only see the bashing comments. Another day you might be in a different mood and choose another "subforum".

This idea can be taken further: make a "subforum" of this kind for every single user. You can let anybody rate any comment and localize the effect of this moderation.

I think this system can be best explained as a topology. Imagine a metric space (plane will do). Every user's point of view is assigned a point in this space. Every comment is also mapped to a point in this space. Now, the position of a newly posted comment is its poster's position - it reflects his point of view. When this comment is rated positively by another user, it moves closer to his point of view and the moderator's point moves (slightly) closer to the comment's position. If it's rated negatively these two positions move away from each other. After several ratings (and several iterations of this procedure), the comment will stabilize.

Now suppose a new user logs in. If his point of view is far from the established point of view of the comment, this comment is hidden from him. The user may actually set something like "vision radius", and he'll see only the comments inside this radius.

Or we may allow the comments' points of view to have "size" or "visibility" that corresponds to its total rating. If a comment is positively rated from all sides, it will be very visible. If, on the other hand, it's positively rated by some users and negatively by others, it would be visible only by users close to it's point of view. Hmm... this part actually looks a lot like the classical system. Never mind.

A good start, but.... (5.00 / 1) (#40)
by chewie on Mon Oct 02, 2000 at 04:47:53 PM EST

This algorithm fails for a couple reasons. The algorithm obviously favors a community opinion over the dissenting member's opinion.

  • If I moderate, my vote will count for very little.
  • If I post, I will likely loose much of my "karma" or "points" just for sticking my neck out.

The algorithm also fails, because it gives no contextual differences to the moderations.

  • I may like the post, but think it's off topic. (do I select '-' or '+'?)
  • The post was written by a 5 year old with insight (bad grammar, awful spelling, bad flow of thought, but good content. '-' or '+').
  • Absolute troll, but hilarious! (Can't stop laughing! Totally off topic! Grotesque! '-' or '+'?)
  • Flamebait (looking for a war, but damned good points! '-' or '+'?)

Moderation is, or should be, a manner in which to reduce the Noise/Signal ratio. That ratio is defined relative to the member reading/posting to a given topic, the formula of which takes content and context into consideration. This algorithm ignores these factors completely, and therefore the Noise/Signal ratio will only be determined by a simplified communal perception of the post.

Conclusion: This algorithm fails.
assert(expired(knowledge)); /* core dump */

Re: A good start, but.... (none / 0) (#41)
by nebby on Mon Oct 02, 2000 at 05:46:27 PM EST

Moderation is, or should be, a manner in which to reduce the Noise/Signal ratio. That ratio is defined relative to the member reading/posting to a given topic, the formula of which takes content and context into consideration. This algorithm ignores these factors completely, and therefore the Noise/Signal ratio will only be determined by a simplified communal perception of the post.

If that's the case, maybe I'm redefining moderation. You have to make a decision to mod that funny troll up, don't you? Well, if the dude is funny and gets modded up by a good handful of other folks (lets disregard the fact that a band of trolls could flood the site deliberately) then I think he deserves to get points. It's not so much signal/noise about being on topic or having bad spelling. It's signal/noise for if people generally like the post or not.
Half-Empty: A global community of thoughts ideas and knowledge.
[ Parent ]

Re: A good start, but.... (none / 0) (#45)
by chewie on Tue Oct 03, 2000 at 03:16:42 AM EST

If that's the case, maybe I'm redefining moderation ...[snip]...It's signal/noise for if people generally like the post or not.

I think the keyword above is definitely "generally". What I didn't post was my initial reaction to the cost system based on popular threads. You've got two side affects because of this, it'll quell a large number of posts to the same thread under the threat that you could loose more points than you're willing to should your post prove "unfavorable." The effect will be to force conversation across a broader range of topics. Not a bad thing if that's what you're going for.

I do think, however, it will make people think about what their posting. My question is whether or not you think dissenting opinions will have a life under this system. Let's say a large number of Windows users become regulars on the forums. A member from a minority opinion group, say FreeBSD, pops in and wishes to make a few valid points as to why FreeBSD is better at <insert favorite topic>. She may very well make valid and well presented points to the <insert favorite topic>, but is wholely ignored or moderated down, loosing her "points" trying to infuence a group of "3l33t Wind0z3 lUs3r5".

The question is of human nature. I would argue that under the above situation, she will be moderated down because her opinion is unpopular, regardless of its content. Popular meaning, "people generally don't like it." Just because someone has an unpopular opinion doesn't mean it's not a good opinion...

I'd say just watch and see what happens. We will all have our opinions of the outcome of the moderation algorithm, but until you've got some "general" use of the algorithm, we'll have nothing to substantiate our opinions with.
assert(expired(knowledge)); /* core dump */
[ Parent ]

(3.00 / 1) (#42)
by zog on Mon Oct 02, 2000 at 06:29:42 PM EST

Maybe there should be a way where one could have the flamebait/insightful/.... stuff, and individual users could put preference on certain types of comments? Just a thought... Otherwise, it looks like a good idea (though communism does about the same on paper too :)
How goes the world?
Good idea, but... (2.00 / 1) (#43)
by mabs on Mon Oct 02, 2000 at 08:29:08 PM EST

Not very scaleable, and chews up cpu time, and prople would have to actually be actively involved, when I go to a news site, I want to be able to read what looks interesting, and continue on.

Useage can be put into a good site then, articles could have an initial interest rating, if 20,000 people visit the site in a given time, and one article is being ignored, there is probably a reason, if no-one posts a comment to an article, they're opinions aren't strong on the subject, or they share anothers opinion.

Doing something on such a large scale every 5 minutes could inadvertantly be a DoS, which is bad...

IMHO it's a bit of an over complicated system, but it might just work.


Re: Good idea, but... (none / 0) (#46)
by Pakaran on Tue Oct 03, 2000 at 11:33:10 AM EST

Go to the site now that it's active, look at all the stuff under 'new ideas', and let me know what you think.

Disclaimer: Yes, I am the Pakaran on that site. Yes, I am biased, but only by all the wonderful things I've seen there.

[ Parent ]

I made a self moderating chat site too (1.00 / 1) (#44)
by reaster on Tue Oct 03, 2000 at 01:02:53 AM EST

I made a self-moderating (user-moderated) chat site too:
<a href="http://www.comptechnews.com/">http://www.comptechnews.com/

No one goes to it, but it has an elaborate scoring system on messages which are posted immediately. Users can also create new topic areas. I just need to make the site look more attractive! I'm not a great computer artist. Its made with PostgreSQL and php.

It was fun making it and basically it works ok. If anyone is interested in it, go take a look. Maybe I will open-source the code so people can play with it. The database code is fairly good, but the php code I wrote could benefit from some better modularization to try to separate the look from functionality. This is the kind of work I wish I could do to make money but I'm sitting here unemployeed :(



Making the Perfect Moderation System | 49 comments (44 topical, 5 editorial, 0 hidden)
Display: Sort:

kuro5hin.org

[XML]
All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!