Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

[P]
Meta: Submission Queue and Story Moderation Issues

By rusty in News
Wed May 24, 2000 at 07:56:08 PM EST
Tags: Kuro5hin.org (all tags)
Kuro5hin.org

feline and several other people lately have commented, both publically and via email to me, about the problems with story moderation. Among the main gripes are the little useless comments like "Neat story" that get posted with votes, and the fact that comments about the content of the story are not distinguished from comments about the form of the story. Yes, it's true, the story moderation system is a new idea, and the bugs are far from ironed out yet. :-)

There seems to be a general desire to talk about this, and I think it is apropos to Kuro5hin itself, since we do have a tradition of occasionally hashing out meta-issues right here on the front page, and so far, I've gotten all the best ideas for the site that way. So I'm going to re-post kmself's posting to scoop.kuro5hin.org here, and open the floor to you. Please gripe, suggest, recommend and thrash out pros and cons to your heart's content. I will be taking careful note of the ideas here.


kmself writes:

Submission queue

Of glaring problems with Kuro5hin, the one bugging me the most is managing the submission queue. IMO it's arranged around the wrong model -- blind voting. Instead, it should be more of an editors' bull room.

Story submissions aren't a popularity contest, they're a dialog, though at a meta-level. This isn't the time for feedback on the content of a piece, but on its packaging, relevance, and appropriateness.

It's a chance to get stuff into the queue, provide feedback to the author, decide where to put things, see if someone can come up with something better, ditch the idea, or offer rewrites. There should also be a way to indicate timeliness of an item and whether or not it should proceed to front page slowly or quickly.

Finally, voting on final submissions should happen in as expedited a manner as is fair. This means that total "go" votes matter less than voting trends and patterns, and that acceptence policies can be tuned to board needs for fresh content, quality content, or changing needs over time. By this I mean that given a relatively small number of votes -- say 20-30, there should be a pretty clear indication of whether or not a story merits posting. The phenomenon of stories hanging interminably in queue should be avoided where possible (memo to Self: look at final interest in stories with long hang times).

This is based on statistics and the law of large numbers, and assumes things like random sampling and normal distributions, which are wickedly violated by reader participation on web boards, but which should still work reasonably well, possibly increasing n with increased participation and post volume.

The sampling may also want to tie in to user "karma", though this isn't envisioned in the same way on K5 as at Slashdot. Some discussion is here in the "What Now" K5 forum. Idea being that folks with high karma are influential in story submissions as well.

What I'd suggest for a re-working of the submission queue:

Story is submitted. In addition to title, topic, and section, timeliness is indicated, say "hour", "day", "week", "month", "quarter", "year", and possibly a posting window. While most of us are used to "run it now" type stories, there are features which may have lower time-sensitivity, or which are appropriate at specific times (holidays, start/end of school year, seasonal, etc.). Inital votes are

  • Post
  • Dump
  • Revise
  • Merge
  • MLP
  • Spam
  • Troll
...with comments attached as appropriate. Post and dump are self explanetory. Revise should require feedback to the author, and while this could be form-driven, I'd think words work well -- "more content", "more description", "use paragraphs", etc. If several authors want to take a topic offline and rework a submission jointly, great.

"Merge" would be a proposal to combine multiple stories. Useful when similar stories are in queue at the same time. Ideally, results of "Merge" suggestions would appear in the related queue items as well.

"MLP" would indicate the story should go to a quickie queue -- mindless link propogation. This should also get a short (one or two sentence) writeup as well as a sell-by date or time window. Ideally, this would be the Herb Caen-o-Matic[1], a bag of items which would be automatically extracted from on a highest-scores, appropriate time-window basis. Could be daily, weekly, or foo.

"Spam" and "Troll" would indicate that a story was not merely unfit for posting, but grossly inappropriate. Probably with negative consequences for the user submitting it. Not to be used lightly.

And again, the discussion should be open, rather than blind voting. Much as you get to see other people raising their hands or speaking out at a meeting. This is a social process, and so far as it's possible to get folks all over the Web in the same space, that's the sense I'm trying to build. If not final votes, then at a bare minimum, comments should be readable. Votes too, I think.

Anyway, that's what's been knocking around in my head, thoughts, critiques, further ideas welcomed.


Notes:

[1] Herb Caen was a columnist for the San Francisco Chronicle, and the last thing which made the paper worth reading. He became my enemy for life by dying of cancer shortly after I succeeded in moving back to the SF Bay Area. Items, items, hoozegotta item?

Sponsors

Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure

Login

Related Links
o Slashdot
o Kuro5hin
o feline
o scoop.kuro 5hin.org
o kmself
o here
o Also by rusty


Display: Sort:
Meta: Submission Queue and Story Moderation Issues | 57 comments (57 topical, editorial, 0 hidden)
The presentation and layout were qu... (3.00 / 1) (#5)
by angelo on Wed May 24, 2000 at 05:18:15 PM EST

angelo voted 1 on this story.

The presentation and layout were quite nice, and follow a logical order. I say post it. -- That is what you want, right? In all seriousness, I think it is a good idea to stick to editorial comments here, and link them to some sort of history button instead of carrying it over to the main story. What is nice about this method is no AC can get "first post" dibs on an article. And I'm certainly all about MLP -- they are fun, a great diversion, and generally topical (that is, if there was a topic) which is a Good Thing(tm).
lowmagnet.org

Rhrm.. ... (4.00 / 2) (#1)
by Inoshiro on Wed May 24, 2000 at 05:24:56 PM EST

Inoshiro voted 1 on this story.

Rhrm..

I do not agree at all with kmself about karma, or about "new" catagories. True, we should make voting 2d (instead of the current 1d number line) to allow for "meta" things ("revise this," "dupe of story"), but the meta votes should not be the same as the posting votes. Your vote should be two things: post/dump, and a vote on the quality. If the quality is low, but the post value is high, perhaps mail the author to encourage a rewrite or similar. With proper catagorization for MLP/Quickies, you need not add aditional catagories as well. Any MLP/Quickies which are put in without proper marking (for automated merging) will probably be dumped, with a note to the author about proper catagories, etc.

Secondly, while it might be fun to envision karma, it's not a positive thing. Everyone should be equal, and everyone should have to vote blind, lest other votes influence the vote. Making some people more equal than others, as well as adding influencing factors beyond the content and quality of the article itself is not good. I'm sure you can see a few problems developing as "lower karma" people go and vote based on what "higher karma" people voted. Karma whores? I don't want them here.

Finally, quickie "nice story" comments are just as valid as a length disertation on the subject. Just because you don't personally appreciate them, and think they contribute only to the noise, authors find such comments encouraging (if a bit light on info). If you want to filter noise, why not have sorting by rating? Then the higher peer-rated ones should be ahead of the noise you eschew.



--
[ イノシロ ]
Re: Rhrm..... (2.00 / 1) (#11)
by Ozymandias on Wed May 24, 2000 at 08:08:40 PM EST

You've got a lot of good points in there, but there are a few things. First of all, I disagree that little "nice writeup" posts should carry over into the discussion. You're right that they can encourage an author, and so need a place.

How about this for a compromise; during the submission process, all comments to the category "style" or "editing" or what have you get bundled up and emailed to the poster. That way, they get the benefit of the criticism/encouragement, but they don't clutter the cue.

On Karma I agree with you, and just to complete the completely reversed order of commenting, I think there's a place for an MLP category; it would clean out some of the trash without discarding the gems.
- Ozymandias
[ Parent ]

Karma, categories, & other forms of MLP (4.00 / 1) (#16)
by kmself on Wed May 24, 2000 at 08:53:50 PM EST

Karma can be implemented well or poorly. Ultimately, it's a grading system attached to a reward system. The key is to make it fair, reflective, meaninful, and accurate. See my comments on moderation systems. A well-designed karma system should share many of these same properties. In particular, boundedness, and responsiveness, though in the case of karma, the system should reflect recent activity rather than be historically cumulative. That is, the convergent property of comment rating needn't hold for the long term with karma.

A well designed karma system should reinforce both the comment and story moderation system, essentially providing for more effective editorial control. The ultimate karma system would allow you to assign your own karma weights to users, essentially allowing you to choose who you feel ought to be able to pass judgement on the things you read. This is an advanced topic. Worth exploring though. Advogato works something like this.

Essentially I disagree with you that everyone should be considered equally. The whole idea of a moderation system is to introduce bias into selection. Preferably it's a shared and fair bias, which promotes quality participation. Life isn't based on equality, and it certainly isn't fair. There should be equality of opportunity -- a person should have the opportunity to be highly rated either in karma or posts. Equality of outcome regardless of actions is a negative though -- your position over time should reflect your contributions.

The alternative is merely a lack of accountability. A problem with Slashdot's karma is that it ultimately leaves a person unaccountable -- with sufficiently high karma they have a "rep" they can't tarnish (not in realtime, anyway). With sufficiently low karma, they're in a hole they'll never climb out of (due in part to perversities of the moderation system, detailed in the comments linked above).

WRT categories -- while I could see adding a rank to a story [1] as well as categorical responses, what we're actually doing is setting up a continuum:

post | MLP | revise | merge | reject
  • Post: means the story is ready to roll as is.
  • MLP: story is worth posting, but doesn't deserve its own page.
  • Revise: intrinsically worthwhile, but needs work.
  • Merge: intrinsically worthwhile, but redundant.
  • Reject: not worth posting.

While the merits of an MLP post can be debated, it may be that one-liners are posted which fit the category. An interest is expediting the queue such that stories are adjudicated quickly. An ability to specify "MLP, as-is" vs. "MLP, revise" might help. While it's nice to think that stories will be appropriately topified, it simply won't happen in practice.

Meta v. substance comments -- I don't think that the submission queue is the right place to start talking about the content of a story. It's the place to get the story to the discussion board. Sure, "nice writeup" is nice to see -- but I know I do nice writeups by now (modesty, never my strong suite). Problem being that "nice writeup" is a perfectly valid thing to say in the submission queue, but noise in discussion. I feel bad for down-rating people who're complimenting me, but I do it routinely (nothing personal, really).


Notes:

[1] One of several possible mechanisms for affecting story placement on the front page.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Re: Karma, categories, & other forms of MLP (4.00 / 1) (#20)
by Inoshiro on Thu May 25, 2000 at 12:16:21 AM EST

I agree that equality always is not valid. However, the current K5 user base is something like 2k users. Right now, fairness and equality is alright. There might be the odd wacko, like that buzzbomb fellow, but on the whole everyone is fair, educated, or at least no openly hostile :-)

Once you have 20k or 200k users, though, it's time for a change. At that point, a trust system should come into play. No, not karma. Karma is an arbitrary accumulation of things. I know I've had 100+ karma on /., and all I did was contribute positive stuff. The problem is that I could easily contribute nothing but negativity, and still have an incredible karma level. Plus, since it's cumulative, people view it as a game. We don't want that.

So it's a trust system, such as advogato's. The problem then becomes how do you plan such a system? How do you add accountability? It's very hard to do, considering the nature of the medium. If it can be done well, though, it will add to our swelling community.

As for the comments, I agree. Perhaps we should allow a meta tag that specifies that the comment is for the author's eyes only (such as a "well done!" comment which others might see as noise). Combined with a proper "meta" voting (as well as the normal "posting" voting), such a system could show some nice benefits. Of course, if it's too complex, people won't use it. Some planning will be required to do this well.



--
[ イノシロ ]
[ Parent ]
Re: Karma, categories, & other forms of MLP (3.00 / 1) (#25)
by CodeWright on Thu May 25, 2000 at 09:20:55 AM EST

I have to say that I agree strongly with both Inoshiro's initial and followup responses to kmself's proposal.

Any system of moderation which gives some forum members the ability to permanently alter the mechanics by which another forum member posts will be abused.

In other words, if someone posts an idiotic story idea on the spur of the moment, by all means, it should be moderated down -- but the poster shouldn't be penalized for every future post, nor should a small clique of posters be able to mutually pro-moderate and permanently improve the chances that any story the clique submits will be accepted out of hand.

Basically, leave the trolls and karma whores where they belong -- on the slash.



--
A: Because it destroys the flow of conversation.
Q: Why is top posting dumb? --clover_kicker

[ Parent ]
Said vs. read (none / 0) (#41)
by kmself on Sat May 27, 2000 at 06:43:21 AM EST

Any system of moderation which gives some forum members the ability to permanently alter the mechanics by which another forum member posts will be abused.

That's not what I wrote. Try responding to the post, not your preformed conclusions.

The verbal token "karma" seems to have been dragged through the mud lately, which is unfortunate, as it's a useful idea. Essentially, "karma" has bad karma. Been talking with Rusty about this, one suggestion is we use the term "Mojo" instead. I kinda like that: "gotta get me mo mojo", "I got my mojo back", "I ain't got no mojo"....

I'm looking for a largely automated proxy indicator for contribution/editorial quality. Permanance is not a good quality. Short-to-mid term persistance is.

Is there something fundamentally wrong with this concept, or is is merely something difficult to implemnet?

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Re: Said vs. read (4.00 / 1) (#55)
by CodeWright on Mon Jun 05, 2000 at 09:31:39 PM EST

I said:

Any system of moderation which gives some forum members the ability to permanently alter the mechanics by which another forum member posts will be abused.

kmself said:

That's not what I wrote. Try responding to the post, not your preformed conclusions.

errr. i thought i was -- in other words; after re-reading your post, i still stand by what i said.

The verbal token "karma" seems to have been dragged through the mud lately, which is unfortunate, as it's a useful idea. Essentially, "karma" has bad karma. Been talking with Rusty about this, one suggestion is we use the term "Mojo" instead. I kinda like that: "gotta get me mo mojo", "I got my mojo back", "I ain't got no mojo"....

Although I'm not particularly a fan of either the word "karma" or the word "mojo" (it probably has something to do with a deep loathing of the 60's & 70's), that isn't the essence of my disagreement-in-principle.

I'm looking for a largely automated proxy indicator for contribution/editorial quality. Permanance is not a good quality. Short-to-mid term persistance is. Is there something fundamentally wrong with this concept, or is is merely something difficult to implemnet?

Actually, I'm glad you made that distinction. I guess that I would have to say that I'm not "morally opposed" to the concept overall -- just that an equitable implementation seems DEVILISHLY difficult to implement (at least, to my satisfaction -- having been on the receiving end of some decidedly ungentlemanly posts by Paul Dunne here [name calling, i think, doesn't have a place in a forum as mature as this one]).



--
A: Because it destroys the flow of conversation.
Q: Why is top posting dumb? --clover_kicker

[ Parent ]
Re: Said vs. read (none / 0) (#56)
by rusty on Mon Jun 05, 2000 at 11:08:40 PM EST

I agree with you. Karsten and I have talked this over, and haven't yet even approached what I think would be a fair and useful implementation of this. I have hopes that perhaps we can come up with something devilishly clever (K's an ex-SAS programmer, so he knows his statistical shit), but of course the key is it make to do more good than harm. Don't fear that we'll go off half-cocked and solve a problem that the site doesn't have, with a solution that causes other problems it didn't have. :-)

I think one of the big keys is to make sure that any sort of persistent ranking accurately reflect some kind of of meaningful metric in the field it's supposed to relate to. Another is that the effects of such a metric should add value to the site, in some way that clearly augments the natural social process; i.e. there is karma in social situations, because if you read stuff, you learn who's who and where they come down on your personal karmic scale. The idea would be to broaden that concept and make it easier for newbies to build up their personal scale, and above all, to not enforce "mechanical karma" decisions on people who can make up their own damn mind, thank you very much.

Like you said, it's a devilishly tricky problem, and one that may not ever have a good solution. Perhaps the best idea is to try to engineer the system so as to not need a "quality memory". In lieu of that, we're approaching the whole question very carefully, and as a sort of long-term research question. I don't expect it to be an issue for quite a while, if ever, so we've got some time to hack on the concept.

____
Not the real rusty
[ Parent ]

Re: Karma, categories, & other forms of MLP (none / 0) (#46)
by kmself on Sun May 28, 2000 at 07:09:31 AM EST

Once you have 20k or 200k users, though, it's time for a change.

On another weblog, it's about 200 users and there's a mess. Turns out there's one catalyst and a few others (half dozen, maybe three principles) who can't ignore him. Turns the whole thing into a flamefest. I'd actually seen a prior board (well, if a library catalog can be called a BBS) where the warring parties were actually the same guy. But then, authentication was really weak -- multiple personalities ran rampant.

Point being -- don't think that K5 is going to stay a cozy community becaus it's "only" got 2000 users. Not bloody likely.

The trust metric is an interesting idea but still not proven, and it hasn't kept flamage out of Advogato. Ask about the "Dimwit" rating sometime.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Submission moderation is an importa... (4.00 / 2) (#7)
by madams on Wed May 24, 2000 at 05:34:12 PM EST

madams voted 1 on this story.

Submission moderation is an important part of how I interact with K5. I wouldn't mind some more complete tools for dealing with submissions. As an solution to the "comments about content vs. comments about form" problem would be a checkbox on the submission moderation for whether the comment should stay with the story when it goes to the main page.

--
Mark Adams
"But pay no attention to anonymous charges, for they are a bad precedent and are not worthy of our age." - Trajan's reply to Pliny the Younger, 112 A.D.

Yes, post this. I don't necessaril... (4.00 / 1) (#4)
by Pseudonymous Coward on Wed May 24, 2000 at 05:57:08 PM EST

Pseudonymous Coward voted 1 on this story.

Yes, post this. I don't necessarily agree with the proposed scheme, but I do agree with the intent. To quote from another of my voting comments:

I agree that the mechanism could probably use a little tuning in order to help separate queue-comments from post-comments. Perhaps having A and B discussion threads for each article, and a "How'd this get here?" link which reveals the voting discussion? I don't know enough Scoop (none, actually) to patch and contribute, but it couldn't be a Herculean task.

I think a solution along those lines is a good start. The intention is to have the ability to "hide" queue-moderation discussion while still keeping the appropriate discussions -- perhaps using kmself's scheme -- open and accessible. Changes to the moderation system itself could be built atop it, but I think a good first step would be to split voting and after-posting discussion into separate subthreads.

At least that way, there's a place for metacommentary, and those who find themselves annoyed by it can ignore it that much more easily.

I like the idea of the extended set... (4.00 / 2) (#6)
by farlukar on Wed May 24, 2000 at 06:10:24 PM EST

farlukar voted 1 on this story.

I like the idea of the extended set of vote types, but it'll be pretty hard to weigh each of them right to get an objective end judgement (that's still either "post" or "dump")

the discussion should be open, rather than blind voting. Much as you get to see other people raising their hands or speaking out at a meeting
Correct me if I'm wrong, but that sounds like having a discussion about whether or not an issue should be posted. What's the use voting "post" or "dump" if most arguments about the content (I really don't believe the arguments will be only about the subject) are already mentioned before it gets posted?
About the raising hands anology - I think it only adds to the "popularity contest" image. "Hmmm I thought of voting 0 but most people think 1, so..." I don't need to see other's votes to make up my mind.

Just to filter the "neat idea" comments out you could dump all voting comments so that they're only use to explain the vote, as information for the poster.
______________________
$ make install not war

Re: I like the idea of the extended set... (4.00 / 2) (#23)
by bobsquatch on Thu May 25, 2000 at 04:43:14 AM EST

I like the idea of the extended set of vote types, but it'll be pretty hard to weigh each of them right to get an objective end judgment (that's still either "post" or "dump")
As I understand it, 'Post' and 'MLP' mean post; everything else means don't post (yet). The advantage of having multiple categories is that it's easier to figure out how to post it (single-story or part of a quickie-glob), or how to revise it (check facts, edit grammar, more/less text, or just dump it). 'Easier to parse' means 'easier to automate.'

About the raising hands anology - I think it only adds to the "popularity contest" image. "Hmmm I thought of voting 0 but most people think 1, so..." I don't need to see other's votes to make up my mind.
Here's one case where other people's comments (not necessarily their votes) would help: when somebody posts a story that looks legit and informative, but is actually wrong. "Albino Alligators Prowl New York's Sewers" may look like a well-researched news item to my sleep-deprived eye, but when some other voter's comment points out that it's just an old urban legend, I would reconsider supporting it.

Blind voting means that every voter will have to figure it out for themselves -- meaning that misinformation more subtle than my example will make it through. It's easy to fool some of the people; it's much harder to do when people are publicly pointing out your errors. Correcting misinformation in a story should happen before the front page, IMHO.

Just to filter the "neat idea" comments out you could dump all voting comments so that they're only use to explain the vote, as information for the poster.
Even better, let the voter choose whether or not his comment should go on the main page. Or, give the voter two textareas, one for the voting, one for the real discussion.

[ Parent ]
Re: I like the idea of the extended set... (none / 0) (#48)
by kmself on Sun May 28, 2000 at 07:25:38 AM EST

As I understand it, 'Post' and 'MLP' mean post; everything else means don't post (yet). The advantage of having multiple categories is that it's easier to figure out how to post it (single-story or part of a quickie-glob), or how to revise it (check facts, edit grammar, more/less text, or just dump it). 'Easier to parse' means 'easier to automate.'

Yes. Reads minds this boy does. The Force strong with him is.

Voting -- comments, if not scores, would be most good.

Discussion -- see "No talking in line".

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

I'm interested in these metadiscuss... (4.00 / 1) (#3)
by raph on Wed May 24, 2000 at 06:15:33 PM EST

raph voted 1 on this story.

I'm interested in these metadiscussions too, probably because I run another weblog site. I agree with most of kmself's points. Here are some further thoughts.

I like the "revise" idea. Authors should have the power to yank their story from the queue for revision, then put in a revised version on the fast track (ie, if the story was already close to posting, only a few thumbs-up votes should be required to post the revision). I think the concept of feedback is especially important.

I'll be implementing something along these lines for Advogato soon. I'm looking forward to seeing how it all turns out!

Revise (none / 0) (#13)
by kmself on Wed May 24, 2000 at 08:24:05 PM EST

Thanks, Raph. After submitting this item, the first thing I wanted to do was revise it to add an argument for options to revise or yank a submission. Then I wanted to yank the whole damned thing.... <g>

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

In addition to "Post", "Dump", "Rev... (4.00 / 2) (#8)
by homeless on Wed May 24, 2000 at 07:36:11 PM EST

homeless voted 1 on this story.

In addition to "Post", "Dump", "Revise", etc., perhaps "Validate" or similar for those stories that might fall under the category of "Interesting if true"...?
----------------------------------
'SYN! .. SYN|ACK! .. ACK!' - the mating call of the internet --Bert Hubert

Fact checking (none / 0) (#14)
by kmself on Wed May 24, 2000 at 08:27:56 PM EST

Fantastic suggestion. I'd considered this in another Scoop discussion, sid=moderation, Kuro5hin style (post ID #3). Factuality and/or validation is an important by often bypassed stage of contributed-content weblogs.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

I think there should be a "hold" qu... (2.50 / 2) (#2)
by evro on Wed May 24, 2000 at 07:51:21 PM EST

evro voted 1 on this story.

I think there should be a "hold" queue for stories that are good but need fixin'. I like the idea of being able to vote a story as MLP. A story voted into the MLP queue would silence all the people who vote down such stories. Then when the queue reaches MAX_QUEUE_SIZE the whole batch of quickies gets posted automatically.

The other thing needed is either the ability to have posts put in votes not show up at all, or possibly not have any commentspace at all in the voting. I find that a lot of times I say my piece in the voting booth and then by the time the story gets posted my interest has waned and I don't feel like participating in the discussion anymore. Of course this may be a bug in my brain rather than the kuro5hin system.

I think the having a spam/troll vote option is also a good idea. I don't know about having consequences for the user, since I don't think anybody would do this with his own account, but I think voting something as spam/troll should require less votes to boot it from the queue. maybe a spam/troll vote has the a weight of -3 normal votes. or something.
---
"Asking me who to follow -- don't ask me, I don't know!"

Restatement of Caen-o-Matic (none / 0) (#17)
by kmself on Wed May 24, 2000 at 08:55:53 PM EST

You've restated the proposal outlined above, though I add a timeliness attribute.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

My views (4.20 / 6) (#9)
by hattig on Wed May 24, 2000 at 08:01:23 PM EST

[this was a moderation comment - by the time I had finished writing it the story had been posted]

Neat story!

This is how story moderation (or should I call it story editing) should be. Editors rewriting the submission to make a better story, people contributing extra information to the editing "room" to be added to the story, - creating a much more content based site as opposed to a neat link site. Open Source Cooperative Journalism to be precise!

If I was to submit a story on a subject - say there was a new OS that I had found, and I wrote a little piece about it and provided a link, then that would be okay, it might get posted, it might not. With editing, people can add links to other similar OS's in development, someone will rewrite it to become more an article about the feasibility of writing these OS's, and then someone can add more about how all the good points of these OS's should be utilised and the bad points dropped. Finally, you could have a 1000 word report on new OS's and the impact they will have in the future.

People will have to be given "editor" status - prominent members or early K5oids maybe. The original poster will be acknowledged, as would the contributors. It would be much more like the quality process that goes into a glossy magazine, with writers, editors, cutters, picture people etc, as opposed to the slighty uncontrolled zdnet style commentary that we get all too much of these days - mostly uninformed drivel.

Now people can take my comment (I GPL it!) and modify it until it is the perfect comment.



Re: My views (3.00 / 1) (#24)
by dr. greenthumb on Thu May 25, 2000 at 06:59:03 AM EST

I like that idead! ;)

However, there will always be people who will mess it up (either by total ignorance or plain evil), rendering the posting uninteresting, full of leet-speak or simply truncated into "m 0VV$ Jh".

I could be avoided, though, by keeping every previous "version" of the posting avaiable as basis for additional editing. If you changes anything of any version, your contribution gets added to the bottom of the list (and perhaps overwrites your previous contributions to that particular posting).

The different versions are subject to user moderation, and after a certain amount of time the highest moderated edit of the posting appears on the frontpage (with credit to the original poster and the 3(?) top-moderated contributors).

Is this a fair idea, or am I full of it?


- 3 out of 2 have problems understanding statistics -
[ Parent ]

Re: My views (2.00 / 1) (#30)
by dash2 on Thu May 25, 2000 at 02:48:26 PM EST

Keeping all the different versions available might be pretty complex after a while. If there are enough moderators at any one time, perhaps people could submit changes to the story for voting, with a suitable time limit, after which votes get counted and the change is accepted or rejected. xxx dave
------------------------
If I speak with the tongues of men and of angels, but have not love, I am become sounding brass, or a clanging cymbal.
[ Parent ]
Got Religion? (3.00 / 1) (#32)
by kmself on Thu May 25, 2000 at 05:37:57 PM EST

I'd try to add to this when I have time, but hattig is seeing through the fog of words I've written to the idea at its root. Collaborative editing is what I'm getting at, and, FlinkDelDinky's comments notwithstanding, editing is not the same as blind voting. It requires feedback, and feedback means seeing and hearing what others have to say, and even being able to change your own mind.

I'm getting a bit of offline feedback on the issue of karma. Most of it largely that, implemented well or poorly, the term has been overloaded by its use at Slashdot and has an overwhelmingly bad connotation to many. Think instead of an aim at identifying people who are willing and able to be good editors. Any performance/feedback merit (aka karma) system is merely trying to automate what may not be a process capable of automation.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Editing (none / 0) (#33)
by kmself on Thu May 25, 2000 at 05:44:52 PM EST

If a story needs rewriting (of which merge is a special case), the simple implementation would simply be for those with an interest in working out kinks to go offline and come back with a fresh submission when the story's ready to go. Providing a collaborative workspace is probably beyond K5's scope at the moment (though it does lead to some interesting possibilities, eh, Rusty?).

Figuring out continuity of a story which has been resubmitted following a rewrite vote is another issue. Probably requires some way to link a resubmit to a prior submission, which means keeping a graveyard of old submits around, at least for some period of time.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Merge options (4.00 / 1) (#10)
by Ozymandias on Wed May 24, 2000 at 08:03:51 PM EST

I like most of the scheme, although I think Inoshiro raised some good points - more discussion down there.

I was wondering about something though - the Merge option could get tricky. How would you work that - if you vote "Merge," you get a form with all the other stories currently in the queue, and select the ones that should be grouped? Some other system? I agree there's a need there, but I don't know as the need is large enough to justify the work involved in getting that system to work.
- Ozymandias

Manual override (none / 0) (#15)
by kmself on Wed May 24, 2000 at 08:30:24 PM EST

...to borrow a phrase from Cryptonomicon....

While a Merge option could be automated, my initial bias would be to simply institute it as a voting item with a free-form field in which other story submission URL(s) could be entered. Moderator would be responsible for marking those stories as Merge as well. KISS.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

one word: (3.00 / 1) (#12)
by Anonymous Hero on Wed May 24, 2000 at 08:15:16 PM EST

Wiki.

In other words, make the stories, then give free, unconditional editing privileges to EVERYONE with a username. Keep a log of who edited when, so that if anything bad comes up then the person can have their account revoked (though this should only be necessary in extreme emergency). Also, eliminate the story queue: All stories go straight on the second they're posted, because they can be edited right there. Finally, let people post DELETE responses to articles, and if there's a certain percentage voting delete, the article's removed.

Re: one word: (4.00 / 1) (#21)
by bobsquatch on Thu May 25, 2000 at 04:10:00 AM EST

If we did the wiki idea (as you've described it, with account deletion as punishment for abuse), we'd have to make k5 accounts somehow valuable or hard to come by. Perhaps we'd need some sort of trust metric for accounts.

Otherwise, if k5 accounts remain as painless to acquire as they are now, nothing would stop me from registering 'bobsquatch1' to hose this story (if I were so inclined). I could then grab 'bobsquatch2' to screw it up again, 'bobsquatch3' when bobsquatch2 is deleted, etc.

[ Parent ]

authentication (2.00 / 2) (#34)
by kmself on Thu May 25, 2000 at 05:48:10 PM EST

Authentication is a fundamental problem of the Net. Ultimately, the solution has be be designed for a system in which authentication (or at least strong authentication) isn't possible. You can't kludge authentication back into the system, you have to account for its lack.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Re: authentication (none / 0) (#38)
by bobsquatch on Fri May 26, 2000 at 03:21:25 AM EST

There's a fundamental point about web communities that many people miss: there's a difference between a person and an account.

I wasn't particularly talking about authentication, which asks "does the account belong to the person?" Rather, I was trying to point out that because each person can create multiple throwaway accounts at k5, removing an account is not real punishment, nor is it even effective at stopping abuse.

A wiki's code of conduct would have to be enforced some other way. One way it could work is by allowing modifications by only certain accounts, and denying priveleges to all other accounts. Suddenly, a particular account has value: you can't make a second account and use that to edit the wiki, because you'd have to get that account added to the access list.

[ Parent ]

person v. account (none / 0) (#42)
by kmself on Sat May 27, 2000 at 06:49:04 AM EST

There's a fundamental point about web communities that many people miss: there's a difference between a person and an account.

Bingo, and probably a better restatement of what I was trying to say.

A scheme like karma or the Advogato trust metric is an attempt to allow privilege to vest to a given account only over time. I looked through the Wikki stuff a bit, it's interesting, but not greatly appealing at this point. Another interesting model, incidentally, is Everything, by the same folks who did Slashdot. Both Everything and Wiki are much more webs than dialogs. There's little or no sense of time or space, only of relatedness of ideas. Pretty cool concept, though much of Wikki seems to revolve on the inherent coolness of being Wikki. If I wanted to revel in inherent coolness, I'd move to California (wait a minute....).

So, what's required is a way to either limit the ability to make multiple accounts representing one person (bloody unlikely as hell), or to make multiple accounts not matter.

Thoughts?

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Re: person v. account (none / 0) (#43)
by analog on Sat May 27, 2000 at 07:52:26 PM EST

Maybe I'm misunderstanding what you're getting at (and apologies if so), but it seems to me that one of the key things you're after is to accumulate power in a subset of readers according to some yet to be defined criteria. My question is, why?

K5 has been built on the idea of everybody having a say, and that given the opportunity to do so, most will do the 'right thing' and drown out those who choose not to. Implementing a system that works to concentrate privilege within a smaller group effectively kills the experiment. Especially since there aren't even any real hints at problems to date (I would say things have been rather spectacularly successful), I'm not sure why it needs to be changed?

I'll grant that some tweaking can be done; some way to allow submitters to edit and resubmit their articles would be fantastic. I think that it can be done with only minor changes to the current process though; I certainly don't think the creation of an 'elite editors' cadre is necessary. Let people have the opportunity to pull, edit and resubmit. I'd be willing to bet the rest will take care of itself.

Maybe the day will come when K5 has to deal with the swarms of trolls that have subsumed Slashdot. I personally think it's unlikely. Slashdot didn't get the way it is overnight; a few people started trolling, others saw that they were getting away with it, and it snowballed. Given that Rusty has no problem unceremoniously dumping that sort of thing, I'm not sure it will ever be a big problem here.

[ Parent ]

Re: person v. account (none / 0) (#49)
by kmself on Sun May 28, 2000 at 07:06:11 PM EST

Maybe I'm misunderstanding what you're getting at (and apologies if so), but it seems to me that one of the key things you're after is to accumulate power in a subset of readers according to some yet to be defined criteria. My question is, why?

42.

I'm beginning to ask myself that. Particularly the way it's going over.

First, the primary objective of the lead-in post is to rework the submission queue, with the primary concerns being that it work faster and produce better submissions. Completely independent of any karmic scheme.

There are some highly time-sensitive posts, mostly concerning DMCA activities in my case, which nobody seems to find out about until five minutes after they've started -- well, often only a day's notice, occasionally a week or two -- which really need to hit a weblog quickly. To date K5 has been far more responsive than /. for these types of issues. The Mozilla privacy (cookies/img) story was another. K5 scooped Slashdot on that story even though it had been submitted to /. first. The flipside is stories that want to die and end up sitting in queue until Rusty puts them out of their misery. I hate seeing things suffer....

The second concern is dealing with topics that need work, either because they're poorly written submissions, or because the duplicate or overlap an existing topic or submission. The current queue feedback mechanisms are very weak in this regard. Inoshiro, CodeWright, and FlinkDelDinky are strongly opposed to this. bobsquatch, hattig, and ramses0 got my religion. I'm not sure why the split here, but I know what I like.

Both these ideas are largely independent of any "persistant consequence metric" (PCM, aka karma). If you'll note, the lead Rusty copied in goes five paragraphs before suggesting a karma-like quantity might be worked in, and continues for another eight without mentioning it again. PCM isn't a requirement for any of the issues above, but if designed properly, could provide some useful features, without generating the animosity it's caused at /.

An exponential moving average over relatively recent (say past 1-3 months) activity used as a basis is what I'm thinking of. An exponential moving average applies a diminishing weight to values in a series, the effect being a heavier weighting of more active This shares many of the properties of a good moderation system -- boundedness and convergence, along with responsiveness to recent behavior, which is crucial for a working incentive/disincentive program. For more information on data smoothing techniques, StatSoft has a good primer. Note that my background in stats biases me toward statistical treatments. An engineer might prefer FFTs or something similar.

One thing it effectively shortcuts is the "rush for the top/bottom" -- people who've intentionally sought very high or very low /. karma. The easiest way to floor or ceiling your rating is to have one highly or lowly rated post. There's no point accumulation, just a weighted average. However, since PCM is based on a weighted average, the more activity an account has the more likely its PCM will vary from "perfection" to a more median level. Ideally, it should be like a good reputation -- something that takes a lifetime of good deeds to get, but can be lost in short order. Well, sort of.

The benefit I see from having a vote-weighting scheme (that's mostly the way I see this), ranking say from 0 (confirmed trolls) to 2 ('5' rated gods) would be to get submissions through the voting queue faster by granting a (slightly) more significant vote to people who've shown an ability to write thoughtful comments. Hand-picked editors with similar privileges would be another way to accomplish the same task, and might be a better way to go. The ideal solution would probably be to pick who you wanted, on an individual basis, to be on your "editorial committee", and be able to grant editorial power on a discresionary basis. This might involve some reworking of the Scoop base code though ;-)

Unlike /., where karma is both very persistant and very powerful, I see PCM as something which changes responsively to current actions to reflect status, and only provides a few minor boosts. Persistant high ratings of stories (a really bad runaway karma feedback loop at /.) not being one of them. I don't think PCM should influence either initial comment moderation or page placement. The moderation system at K5 is sufficiently responsive that good (or bad) posts will find their level quickly.

A flipside attribute might be to restrict viewing of low-rated comments to people with low ratings. Based on the idea that most of the reason for "tagging" a site is to see your own work, the lower a reader's own rating, the higher your minimum filter post might be set. It's got some really strong Big Brother overtones to it, but think of what your impression of /. were if your browsing threshold were restricted to 3+? Turns into a whole 'nother world. Likewise, K5 could default to a 'Minimal', Highest first', 'Threshold =3' for anyone with an average comment rating of less than 2. If you manage to post average content or better, you can see everything (if you want).

An advantage to any kind of PCM is that it gives a certain value to longer-lived accounts. You don't want the value to be too high or too hard to achieve or you get whoring, but a moderate ramp should prevent hit-and-run attacks. I've seen these elsewhere. Sites prone to them get ugly.

Various boolean attribute story and comment ratings -- I'd like to see them implemented, but on an advisory basis initially. Again, these might be related to the PCM -- only those with an average or higher rating could use these controls. After we get some history on usage patterns, automated disposition of troll and spam content could be instituted. Incidentally, for a number of copyright and liability reasons, any such mechanism should almost certainly be automated rather then be a manual process controlled by the K5 admin.

Finally, a third option for a bounded PCM is to use it as a multiplier. Any action taken by an individual is multiplied by the PCM value. The downside control is more crucial IMO than the upside -- you want trolls and such to have little access to these controls. You get this by giving them a multiplier near zero. Someone who's truely godlike might get a value near 2. Most of the world floats around 1. That's not a huge spread from mere mortal to god, but nefarious evildoers are effectively crippled. Might make things work a bit more smoothly. Of course, it's a bookkeepping and programming nightmare....

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Re: person v. account (5.00 / 1) (#54)
by analog on Thu Jun 01, 2000 at 01:48:48 AM EST

Hmmm. Hope you're still following this; got busy and forgot to add it to my hotlist.

I guess (hope ;) I've got a better idea of what you're after now. For the record, I agree with a lot of what you are saying. The devil is, however, in the details.

That some sort of editing capability is needed for the submission queue seems a no-brainer. However, I'm not sure I'm happy with any idea that kills blind voting. I know you don't seem to like it, and you've made some good arguments for doing it otherwise in other posts. It does appear to work well, though. If it ain't broke...

If there's a weakness, it's that over time it seems to break down; given enough time to sit around in the queue, 'iffy' stories that will generate little or no discussion often get posted. A shorter time limit for stories to make the posting threshhold could fix that. If Rusty wanted to get really funky with his programming, he could set something up (say) such that it takes more positive votes vs. negative votes to keep a story alive as it ages. Point being that I think minor tweaks are more in order than an overhaul.

As far as editing goes, I'm still not sure how to go about implementing that (although I have some ideas of at least a couple of different ways it could be done). I do feel strongly that last word goes to the author; if he/she wants to post something as written despite the best efforts of <insert editing body here> to the contrary, that's how it should go. I think over time you'd find that poorly written stories would become less likely to get posted, and authors more willing to accept suggestions. Even without the ability to edit posts, we're seeing some movement in that direction.

The whole PCM issue (I think dropping the term 'karma' is probably a good idea at this point) is problematic. I think I understand why you want it now. I think a good idea here might be to take a page out of Ray Dolby's book (stay with me here ;). Dolby noise reduction is based in part on the principle of 'least treatment'; do just enough to fix the problem, and leave the rest alone. So you could have your PCM weighting, but only have it usable on stories where it really matters. Time sensitive stories could be boosted by those who have the higher ratings. Perhaps editing privileges, but only after sufficient moderation queue comments/scores had marked it for editing. That sort of thing.

I'm operating on not much sleep and my thinking is getting a little fuzzy, so I'm outta here for now. I definitely think this deserves more discussion; most people seem to think that there are ways to make things better, if we can just agree on what they are. ;)

[ Parent ]

Ok, you stumped me (none / 0) (#35)
by kmself on Thu May 25, 2000 at 05:48:40 PM EST

What's wiki?

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Re: Ok, you stumped me (3.00 / 1) (#36)
by FlinkDelDinky on Fri May 26, 2000 at 12:37:19 AM EST

Okay, I'm really not that knowledgeable about Wiki's but as I understand them they are the original collaborative web logs. I think they were first implemented with and for SmallTalk.

I've never participated in one but to me they look a lot more collaborative and a lot less structured and organized then /. style web logs.

Here is a place that seems to have some relevant info, scroll to the bottom and check out the faq

[ Parent ]

Wiki (none / 0) (#52)
by kmself on Tue May 30, 2000 at 05:07:53 PM EST

First note that Slashdot is both a weblog and a Wiki, if you include Everything.

The general idea behind a Wiki -- rolling through an idea space -- is pretty cool. There's a strong argument for topical stuff as well. I think the best thing to do would be to tie the two together somehow, though how is hard. Note that /. does this by linking Everything keywords in posts to the Everything topic for the word.

The mutability of a true Wiki isn't something I'm quite willing to accept, though I'm also not dismissing it out of hand.

The worst thing about the Wiki seems to be that an awful lot of the material is self-referential -- look at this, it's so cool, it's a Wiki, and, hey! I modified your content -- me too!!.... Etc.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

First wiki (none / 0) (#57)
by Sunir on Wed Dec 13, 2000 at 11:30:19 AM EST

The first wiki was implemented by Ward Cunningham in Perl.

"Look! You're free! Go, and be free!" and everyone hated it for that. --r
[ Parent ]

Change the 'merge' idea into something else (3.00 / 1) (#18)
by Anonymous Hero on Wed May 24, 2000 at 10:32:53 PM EST

or at least allow for the previous (but current) discussion to be linked to directly, so that it can be 'folded in' rather seamlessly. Maybe as an 'update' or 'related', as the case may be. --sugarman--

Re: Change the 'merge' idea into something else (none / 0) (#19)
by Anonymous Hero on Wed May 24, 2000 at 10:36:58 PM EST

Never mind. See #10, above. (Relatively speaking)

[ Parent ]
Will the real K5 please stand up. (3.90 / 7) (#22)
by FlinkDelDinky on Thu May 25, 2000 at 04:40:34 AM EST

Kmself has a lot of interesting ideas. Unfortunately I think they're bad ideas.

The Spam and Troll selections will be abused by trollers using throw away accounts. I suppose since you're going to suppress participation by using karma this could be controlled.

Merge sucks too. If I'm sick of a subject I'll dump it. Jeez, what's the big god damn deal if you get a bunch of mp3 stories? It doesn't matter of you coral them all together. If the community votes them to the front page so be it. The submitter thought it was different enough to do a separate submission. If you don't agree dump it, maybe tell him to post under an existing article if you want, but I think that's kind of stupid. Merge is an unnecessary and stupid addition of complexity.

Open voting is bad. I want the individual to decide on the value of the article. Let people learn to form their own opinions and think for themselves. Let them learn when to abstain from voting. If you do open voting the weaker voters won't learn to become strong voters, they'll just fall in line with an elite cadre of users, if only on a subconscious level. K5 wants to know what the individual thinks and from that we learn what K5 as an organism thinks and how we as individuals are different and similar to each other and to K5 as a whole. I find this extremely valuable and its genesis is closed voting. K5's greatest strength is its ability to value the individual.

I'm unable to fully express my intense (but entirely reasonable) dislike for karma weighting the vote process. I posted a comment that was rated a three and in the post I asked a question. The post was worth a three. Another user posted a short reply with a valuable link, that post was rated a four, but really worth a two or perhaps a generous three (the link was to a mildly anti-Linux statement). I replied to that post with little more than a link to a related less anti-Linux page, and it got a 5 rating, worth maybe a one at best.

The point being that karma isn't perfect. I see no reason why you think it's okay to allow people to vote karma but not submissions. Why make K5 a republic? K5 is and should remain a pure democracy. That is it's strength!!! K5 is a reaction to the /. republic remember? You want to go to war with the K5ers, you want to minimize and suppress their full participation as equal (and deserving to be equal) members and creators of the K5 experience.

And as for the little one liners, keep'em. BFD, a few on-liners, some snipes, some encouragement, they all wind up at the bottom with 0's. If that was what a K5er wanted to contribute, so be it.

I'm also against hattig's 'open source' editing. Remember, we have commenting folks! Commenting is for expressing new ideas, adding new knowledge, disagreements, and corrections.

I agree that the queue needs a bit of work. I had warmed up to the vote by category 'race' concept but now I dislike it under the grounds of unnecessary complexity and time consuming to boot.

For voters I'd like to see:
frontpage
abstain
dump
edit

Edit would be a vote that forced the article to remain in the queue. If we have a timeline (which may be a good idea) then that would be put on hold as well.

for submitter's I'd like to see:
yank
submit re-edit

I'm not sure how to handle prior voting upon 'submit re-edit'.

This post has be on the harsh side. I just want to say I enjoy Kmself's articles and participation. But I kinda have to say that cause I forgot my asbestos suit and I've run out of flamethrower fuel...

Re: Will the real K5 please stand up. (4.00 / 1) (#26)
by hattig on Thu May 25, 2000 at 09:51:38 AM EST

I like my ideas - it makes K5 less of a Slashdot and more of a something that hasn't really been done before. There are weblogs everywhere, where links are posted and people comment. This is the first real chance for people to submit articles, and then for other people to add their own knowledge and create a good article. Then there can be discussion of it afterwards!

I had the idea of editors - people who are responsible within Kuro5hin - who do the actual story editing, be it correcting spelling mistakes and grammar (as it would be initially) and moving onto integrating interesting and relevant voting comments into the story itself. Someone could post an interesting article, but someone could present arguments for and against points in the article, or refute elements of the article. These comments would have been previously hidden in the comments, this allows errors and changes to be caught before story posting (printing).

I agree with the voting options though. No actual scores for your vote, just nice options to select. As a story submitter I wouldn't mind being able to make edits to the story before it is posted, or yanking the story if I suddenly find out it contains incorrect information or confidential information.

Rusty, how long until we can theme the look of Kuro5hin for our own tastes, even if there is only a limited selection of styles. Surely easy to implement because of the stylesheets?

[ Parent ]

Re: Will the real K5 please stand up. (5.00 / 2) (#37)
by ramses0 on Fri May 26, 2000 at 01:13:16 AM EST

Sorry I haven't posted in a while, I've been trying to be good while at work and stay away from the weblogs ;^)=

But..... scoop is open source, and if you feel like implementing theming, or community editing, go right ahead and do it. This is known as "put up or shut up", meant in the best possible way.

I tend to agree with the previous poster, that k5 is uniquely democratic... if you really care, express your views by moderating.

I agree that K5 needs some extra tools in order to help out in the moderation zone: "carry-over" vs. "trashing" of vote comments would be an easy, nice option to have.

Email summaries of story information to the submitter would also be very nice (ie: these were the final vote tallies, here were all the comments made) .

Whoever came up with "submitter" options of "yank", and "edit/repost" was pure genius. And instead of the current "editors@kuro5hin.org" system for editing a story, howabout an "editing queue", so that users can say "Admin->Edit submissions" instead of "Admin->Moderate submissions".

Make the "edit" queue separate but equal to the moderation queue, only difference being that stories in the edit queue never get posted (ever!), but people can still blind-vote on them. Maybe add a discussion area (like a real k5 story), and whenever the submitter (or somebody else) feels that the story is ready for prime time, they can go submit it to the real moderation queue, or let somebody else submit the story to be moderated.

Anyway, time for an example:

  • Bob thinks to himself: "i'd like to discuss why mp3 is so successful"
  • Bob posts something to the edit queue: "I want to discuss mp3, but first i need some help finding links"
  • a nice group of k5 people post links for bob in the edit queue, and have a helpful conversation using the regular scoop discussion engine (blind voting could be optional??)
  • armed with these new links, bob runs the "yank" command on the article in the edit queue, then submits a new writeup (which he wrote using information he gained from the edit queue)... but he submits this writeup to the moderation queue.
  • the story gets moderated up, and makes it to the front page.
  • Yay. We get
    1. community editing
    2. hopefully "merge-able" ideas from other users
    3. hopefully less crap/spam (because of the completely meta-quality "edit" conversations, on whether to post it or not)
    4. all using the exact same interface and codebase that we are all used to

/me is tired, but I think this should make some sense.

--Robert
[ rate all comments , for great justice | sell.com ]
[ Parent ]

Shared vision (none / 0) (#44)
by kmself on Sun May 28, 2000 at 04:23:58 AM EST

Ramses, you're doing a really good job of saying what I didn't seem to be able to. Thanks.

Yes, the whole point is to make the submission queue an editing mechanism rather than simply a voting mechanism. The inspiration for the idea is, naturally, the open source development process itself, which involves collaboration, discussion, interchange, and a meeting of minds, rather than a simple up-or-down voting on code submissions.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Re: Will the real K5 please stand up. (2.00 / 1) (#27)
by CodeWright on Thu May 25, 2000 at 10:06:36 AM EST

hear hear!



--
A: Because it destroys the flow of conversation.
Q: Why is top posting dumb? --clover_kicker

[ Parent ]
You called? (none / 0) (#50)
by kmself on Sun May 28, 2000 at 08:25:21 PM EST

Kmself has a lot of interesting ideas. Unfortunately I think they're bad ideas.

My comments are intended as a discussion point. For the most part I'm concerned with speeding the submission queue and improving the quality of stories. Introducing a "karma"-like system is incidental to this, though it could provide benefits. You have a strong preconceived notions that any such system is identical to Slashdot's implementation -- I certainly hope not. I also find your arguments regarding submission queue voting singularly unpersuasive.


Merge sucks too. If I'm sick of a subject I'll dump it.
If a story's not worth posting, the proper vote is "dump", not "merge".

If you don't agree dump it, maybe tell him to post under an existing article if you want

That's precisely what I'm looking for.

If a post raises good issues but closely duplicates existing content, suggest that two authors work together rather than submit competing content. If the post properly belongs as discussion underneath another (the followup Meyer article IMO should have been), note same.

Weblogs tend to compartmentalize convesations under many different pages, instead of combining similar threads. This is particularly true with rapidly (r)evolving front-page content. If two conversations are one logical unit, provide the support to make it so. "Merge" is not a functional proposition, it's an advisory one saying "merge this story with foo". The effect is similar to dumping the post, but the message is that the information is inherently interesting.

Three counterexamples are: Usenet, the InfoWorld forums (typical example), and IWETHEY[1]. Rather than Slashdot's rapid-fire post, rant, and retire mode of discussion, all three of these formats allow a topic to "live" as long as there is active interest in it. Usenet does through by use of newsgroups which somewhat focus dialog, and threading, which maintains topical integrity. Wiki and Everything are other interesting examples based very much on the web metaphore.


Open voting is bad.

We've clearly got different views here. Part of the problem is that we're thinking of different activities.... First of which, the process is editing, not just voting. Deciding what's worth posting is a part of the process, but there's much more to it. From Merriam-Webster Online:

editing: 1c. to alter, adapt, or refine especially to bring about conformity to a standard or to suit a particular purpose: carefully edited the speech
Without belaboring the obvious [2], both hattig and ramses0 did a good job of describing the difference between a simple submission queue, and the option of interactive editing.

Simply, the model is that of the open source, "Cathedral" development model itself. And this requires open voting -- at the very least open commentary and interchange. There are a number of interesting studies of decisionmaking in light of the free software phenomenon. Most research finds better decisionmaking outcomes from groups than from individuals. One study (Sniezeck, 1989) evaluate five types of group decision techniques [4]:

  1. consensus: face-to-face discussion resulting in a single group answer.
  2. dialectic: group discussion of decision bias factors.
  3. dictator: best member's solution is selected.
  4. delphi: anonymous responses provided in "rounds" until consensus or stability is reached..
  5. collective: interaction prohibited, individual judgements are aggregated to form "group" response.

The context was five college students asked to solve a quantitative practical problem. In the study, method five provided the worst results. Each of the first four was better, but the "dictator" method reduced absolute error by a factor of three over any other. I'd prefer to call this a meritocracy, though Linus Torvalds occasionally refers to himself as a "benevolent dictator". Key to success is that vesting of the "dictator"'s power is the vote of the group -- the only sway the dictator holds is her ability to come up with good results, there is no other vesting of power.

General results of group decisionmaking:

...for brain teasers [and similar logic puzzles], groups usually outperformed individuals, but...the best member of a group, working alone, tended to do better than the group as a whole.

...brainstorming was more effective when ideas were generated independently and later combined.... [T]he best way to generate solutions to a difficult problem is by having several people work on it independently and later share their ideas.

If you do open voting the weaker voters won't learn to become strong voters, they'll just fall in line with an elite cadre of users, if only on a subconscious level.

I far prefer to think that by seeing the thought process involved in deciding merits of a story, people may actually learn to walk through the steps themselves.

K5 wants to know what the individual thinks and from that we learn what K5 as an organism thinks and how we as individuals are different and similar to each other and to K5 as a whole. I find this extremely valuable and its genesis is closed voting. K5's greatest strength is its ability to value the individual.

Bollux.

This is encouraging people to work in cloistered isolation, individually propogating various sets of mistakes. Better to employ the power of the individual in addressing the strengths or weaknesses of a story, and working to either craft it into something better, or ditching it rapidly and painlessly (well, with some help from tweaks to the queue discrimination algorithm).

Why make K5 a republic? K5 is and should remain a pure democracy. That is it's strength!!! K5 is a reaction to the /. republic remember? You want to go to war with the K5ers, you want to minimize and suppress their full participation as equal (and deserving to be equal) members and creators of the K5 experience.

First, neither Slashdot nor K5 are a democracy or a republic. They're collaborative content engines. /. happend to pick a particularly broken collaborative rating scheme. It worked fairly well with 40 moderators, passibly with 400, but didn't scale beyond that. K5 is principally the same fundamental weblog with a different moderation scheme, no filtering mechanism, and a few different viewing options.


Good Citizens...

You write:

The Spam and Troll selections will be abused by trollers using throw away accounts. I suppose since you're going to suppress participation by using karma this could be controlled.
And:

I'm unable to fully express my intense (but entirely reasonable) dislike for karma weighting the vote process. I posted a comment that was rated a three and in the post I asked a question. The post was worth a three. Another user posted a short reply with a valuable link, that post was rated a four, but really worth a two or perhaps a generous three (the link was to a mildly anti-Linux statement). I replied to that post with little more than a link to a related less anti-Linux page, and it got a 5 rating, worth maybe a one at best.

You've got two conflicting thought lines going in this little ramble of yours. The first is that the individual matters, and that the sanctity of the individual's thought process isn't to be violated. The other is that there are all these other individuals out there who abuse and troll the system, or who grant inappropriate scores to posts. So which is it -- do individuals get it or don't they? Or do you want a democracy of good democrats only?

There are some people who are good at identifying submissions, there are some people who aren't. There are some people who are good at creating good submissions, at editing, at commenting -- and those who can't. Karma, mojo, juju, whatever you want to call it, is based on a couple of basic assumptions:

  • People have different skill levels.
  • There's a rough association between certain skills. For example, writing good comments, and good editing skills.
  • If you can come up with a good way for evaluating ability, with emphasis toward recent trends, and with a bounded valuation (eg: 1-10, 0-2, 1-5, etc.), and you work out a way to actually apply this value to things such as comments, comment moderations, and submission moderations, you might have something useful.

Your concerns appear to be these [5]:

  • Trolls/spammers could abuse the troll/spam controls to rig the submission queue.
  • Comment moderation could "permanently" reduce a poster's karma to some ungodly low level (actually made here by CodeWright, but I see the same sentiment here).
  • People don't moderate posts (yours or others) the way you think they should.

Let's ditch the word "karma" for a moment and coin "persistant consequence metric", or PCM. "Persistant" (but not permanent) -- the tool causes actions to play out over a period of time. "Consequence" -- the tool is reflective of past actions. "Metric" -- the tool attempts to provide a useful measure. These exist all around us -- personal reputations, grades, workplace evaluations, salary, insurance ratings, credit risk. You name it.

The specific intent is to address your concerns -- that trolls and spammers can't abuse the system, and that those who do contribute meaningfully can have their input reflected. This is very similar to the concept used by the Google search engine, which uses both links and linking-page importance to rank hit relevance.


I'm also against hattig's 'open source' editing. Remember, we have commenting folks! Commenting is for expressing new ideas, adding new knowledge, disagreements, and corrections.

...and editing is for coming up with something worthwhile to comment on. The idea of the edit process isn't to comment on and annotate an article, it's to improve the quality of the article. I'm currently looking at the Gerald Dwyer story (FRB economist paper on free software) in the submit queue. The paper is actually (to my econ-biased background) worthy of a story, though the writeup could use some help. It's had over 110 votes, and appears near the posting threshold, but has been sitting in queue for a while.


Notes:

[1] Incidentally, spun off of InfoWorld's forums by the user "community" after several years of editorial abuse (though the columnists were largely good), and a really bad forums software choice by IWE. A case of "community" up and walking away.

[2] Pity, as I seem to revel in it.

[3] ...and please do me the favor of ignoring the fact that I'm trying to browbeat you into submission....

[4] Discussion, examples, and quotations on group decisionmaking behavior drawn from Scott Plous, The Psychology of Judgement and Decision Making, McGraw Hill, New York, © 1993, ISBN 0-07-050477-6. Particularly Chapter 18, "Group Judgements and Decisions", pp 205ff, discussion the work of Janet Sniezek, Becky Henry, and Gayle Hill. This book is very interesting reading and very highly recommended. Decisionmaking traps, including groupthink, are also addressed.

[5] Note that I've posted asking for other feedback on how specifically the Slashdot karma system fails.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

More comments (3.70 / 3) (#28)
by Rasputin on Thu May 25, 2000 at 12:04:17 PM EST

After reading all the other (currently available) comments, I have a couple of my own.

1. I agree something "karma-ish" is probably not a bad idea, but a lot of care needs to be taken with the implementation. The outline from kmself seems like a good starting point. The biggest problem I see beyond the points already mentioned is that a person's ability to comment on technology is not necessarily related to a person's ability to comment on culture. Unless the rating is tied not only to the quality of a person's posts but also to that person's ability to refrain from posting when that is most appropriate, I don't see that we would gain that much.

2. My understanding is that the comments area in the voting queue is for comments on why a story should or should not be posted (okay, I too have used it for commenting on the subject as well ;). If those comments are dropped when the story moves to the front page, we would see 2 benefits: All the "Neat! Post it" comments would disappear from the front page (a good thing (tm)) and it would reduce the amount of discussion going on in the voting area and move it to the front page where it belongs.

3. With the story discussion removed from the voting area, leaving only the meta-discussion, we could then allow viewing the voting discussion without viewing the current vote totals. I realize this is a less than complete solution to the "lemming issue", but it would allow us to see comments that could legimately sway a vote from a 0 to +- 1 (or from +-1 to 0). It will probably still be abused (people putting in what they voted with why they voted), but I think overall this will still be more benefit than pain.
Even if you win the rat race, you're still a rat.

comments on comments (4.00 / 1) (#29)
by maphew on Thu May 25, 2000 at 02:21:11 PM EST

Whoa, synchronicity happens. :P Last night after struggling with insomnia for an hour I started writing a story. I was (am) having troubles whacking it into a shape I admire, so I tacked a paragraph onto the end which looked something like this:

Please don't vote on this article right now - I know it needs work and refinment. Rather, comment on the style and form. I'll crawl back into my cave, lick my wounds, bang out something which shows craftsmanship and repost it.

+++

I agree that moderation/voting comments should be on a seperate page from the front page comments.

I couldn't care less what so'n'so voted on a story. It's usually more than evident from the comment itself. I see no reason not to do blind voting.

I like the category ideas 'post', 'revise', 'MLP', 'dump'. I wonder how much 'validate' would be abused: story slams linux, linux is god, story sucks/must not be true => 'not validated'. I do like the concept though. 'Merge' could be interesting. 'Spam' and 'troll' are unnecessary (see 'dump').

I also like the idea of a karma or a trust metric but have strong concerns over the implementation. I don't know if it can ever be computerised effectively. It echoes the recent conversation on the difficulty of defining ethics.

Rasputin's point on making a distinction between a poster's ability to comment on technology from their ability to comment on culture is an important one.

+++

Okay so there's nothing in this post which hasn't already been said. Think of it as a vote on the expressed ideas.

cheers,

-matt

Some thoughts, in no particular order... (3.20 / 5) (#31)
by analog on Thu May 25, 2000 at 03:20:51 PM EST

Forget karma, at least for now. It may enable some neat things, but it kills the feeling that everyone has equal potential for making a valid comment. As well, I'm not sure what problem K5 has that karma would address. Perhaps more discussion is needed, but I think at the present time it would do more harm than good.

Leave submission voting blind. It works very well, and gives everyone a chance to be honest with themselves about how they feel. It ain't broke; don't fix it.

An option to give authors the ability to edit a post in the submission queue would be nice. I'm thinking in terms of an extra voting option along the lines of 'needs work'; enough of these votes and it gets dumped from the queue with a notification to the author that it needs work before hitting the front page. Putting the post in a holding area along with the comments would be nice so the author can see what people felt was wrong with it.

It's my personal belief that 'community editing' is a disaster looking for a place to happen. Give the authors the ability to edit & repost, and the community can say what they'd change in the submission comments. If the author agrees with proposed changes, he/she can edit appropriately.

It might be a good idea not to carry submission comments through when a story gets posted. That would tend to keep them focused on things relevant to whether or not the story should be posted (yeah, I'm guilty too ;). This is especially true if we get a 'good article idea, post needs work' option.

just my two cents (3.00 / 1) (#39)
by jeezuzz on Fri May 26, 2000 at 04:36:48 AM EST

One idea I had for a similar weblog system, was to have No Karma for individuals, but every member login would be issued a number of points (say 5 for example) where you could moderate + or - on posts. Extended rules could be - not allowing to post on a page you have moderated on during that session, so you could not moderate down someone you disagree with and vice/versa. Another idea was to have three agreements too issue a point to a post (+-) that would a give a point system more weight and less of a chance for one person to abuse the system. So in other words, it would take three moderators to + or - a post. As far as article submission is concerned, I like many of the ideas I've heard. but I dont have really anything else to add here that hasnt already been said. I think the one thing to keep in mind is that there is never a perfect system, its just kinda going with the best system - that most poeple can agree on - and just being open to new ideas, and being flexible and willing to change as a community grows. Whats right to begin with may be quite unsatisfactory two years from now.

Post moderation vs. submission editing (none / 0) (#40)
by kmself on Fri May 26, 2000 at 07:56:45 PM EST

The problem at K5 is less moderating posts than figuring out how to make the moderation queue work a bit better. That's what I'm steering at.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

"can't moderate and post" is *bad* (5.00 / 2) (#51)
by sab39 on Tue May 30, 2000 at 02:17:47 PM EST

I'm convinced that Slashdot's restriction that you can't moderate and post in the same discussion has caused more problems than it has solved.

A particularly grievous case of poor moderation was the recent interview with RMS, where 3 or more of the questions that got submitted to him used the term "open source" or "oss". Of course, almost anyone who has even *heard* of RMS would know better than to use either of those! And yet three of these were among the top 10 moderated posts? How is that possible?

My theory is that pretty much anyone who had ever heard of RMS had at least one question they wanted to ask him (well - wouldn't you? I did), and so they posted on the story - leaving the moderation to be done by people who didn't know who he was or what he stood for.

Remember, by limiting the amount of moderation one person can do, you're *already* protecting against abuse by ensuring that no one individual has the power to make a significant change to the discussion unless others agree. So the amount of self-promotion (and opposition-suppression) any one person can do is highly limited. Stop them from moderating their own posts, and they're even more limited.

Stop them from moderating any story they're interested in enough to post on it, and you eliminate the people who are by definition best qualified to moderate intelligently, and you're left with... well, slashdot moderators.

Stuart.

PS sorry for the slightly offtopic rant - but there's nowhere on Slashdot where it would be even remotely on topic. At least here there's a chance to *not* make the mistakes /. did.
--
"Forty-two" -- Deep Thought
"Quinze" -- Amlie

[ Parent ]
Karma dogma -- what went wrong at Slashdot? (none / 0) (#45)
by kmself on Sun May 28, 2000 at 06:51:53 AM EST

There are clearly some problems with the way karma worked out at Slashdot. Among these are trolls, spams, karma whores, bitslapping, and some possible "adjustments" of karmic values by Slashdot admins. What else?

Give me an earful.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.

No talking in line (none / 0) (#47)
by kmself on Sun May 28, 2000 at 07:16:55 AM EST

One (less combative) issue that's come up several times is what to do about discussion while a story is in queue. The usual suggestion is "let the comment poster decide whether or not these carry to the discussion page". I'm don't particularly care for this, though I'm not going to fight it if everyone else wants it.

The issue though is this: stories in queue are being groomed for front page. They're not there yet. The queue isn't set up for discussion, it's set up for an up/down vote, and possibly (hopfully) editorial kibbitzing.

"Discussion" in queue really amounts to monologues -- before you have a chance to see what other folks are talking about, and at a point it's not quite appropriate. Among other things, you tend to end up with a lot of duplicated content. Threading isn't established yet (though this could change). It's just not quite right.

My preference would be that the queue is for editorial comment only. Story discussion happens after a story is posted.

But I'm flexibile on this. Really, I am.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.

some comments (4.00 / 1) (#53)
by Paul Dunne on Wed May 31, 2000 at 12:30:39 PM EST

Karma and rating are both bad. One man's meat is another man's poision, etc. Trolls, spam etc should be deleted; other comments should stay up. We should focus on how to keep bad comments to a minimum, rather than working out ways of circumventing them. That can only be done, in my view, by having a cohesive membership that just doens't do "dumb stuff", and that comes done heavily (via e-mail) on new visitors that do.

Story moderation as is works, but kmself is right, there needs to be more of a focus in the comments on why one is voting as one is, and these should not carry over into the discussion when/if the article is posted. These comments should be purely about *whether* the article should be posted, if the decision is, "no", they should give some idea of what is wrong with it, and perhaps suggestions for improvement. Editing is different. Editing involves mainly proof-reading, and perhaps a "pre-mod" discussion.

Blind submission voting is a good thing. It works, don't change it.
http://dunne.home.dhs.org/

Meta: Submission Queue and Story Moderation Issues | 57 comments (57 topical, 0 editorial, 0 hidden)
Display: Sort:

kuro5hin.org

[XML]
All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!