Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

[P]
Voting, the story queue, and thoughts on K5

By Signal 11 in Meta
Mon Jan 29, 2001 at 03:30:07 AM EST
Tags: Kuro5hin.org (all tags)
Kuro5hin.org

Kuro5hin is growing, and with it is a crush of new users and new stories being submitted into the queue. Slashdot receives hundreds of such submissions a day. Kuro5hin routinely exceeds ten. So why am I complaining? Because I'm the one doing all the work! Read on for some solutions.


I have a proposal to solve the queue problem. Many of you have probably noticed that the increasing thresholds have made stories sit in the queue longer - I've seen stories in the queue approach 10 days before being posted (or buried). This is simply not acceptable.

My solution is to set the thresholds to a static amount, and instead vary the percentage of the total users who see the story in the queue. When you first view a new story in the queue, kuro5hin would assign you, say, a 75% chance of being able to view it. If you can view it, you can vote on it. Otherwise, it's hidden from you until it is posted into the database. This way, we can each moderate maybe 5 submissions each day on average. We could, of course, always see our own stories.

There are some other related changes I've been mulling over - one of which is removing topical comments from stories while they're in the queue. Before you jump to conclusions, I think it is necessary for several reasons - often a story is not posted in which posters spend a lot of time composing a good reply. I don't think it is fair to them to have their comments "eaten" by the system, and limiting to just editorial would provide additional focus on the story's mechanics instead of the content, instead of whether or not people agree with the story submitter.

Another idea I'd like to bring to the table is to formalize a way of getting regular interviews (round tables) on Kuro5hin. Perhaps taking the top stories of the week and then posting a followup from experts in those areas and doing some Q&A. Kuro5hin has a focus on culture instead of just news, and I would like to keep moving in that direction by revisiting top stories in some fashion.

My last idea (and I promise, it is for atleast a week!) is an expansion on a pattern I've noticed - Kuro5hin articles often are related to previous Kuro5hin articles. Often it is a different approach, but the context is the same. What I would like to see is a more intelligent "What's Related?" linking those other articles. It would need to be automated, as story authors won't use it otherwise. I have no ideas on how such a thing might be done, so comments from people familiar with Scoop are appreciated.

Sponsors

Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure

Login

Poll
Is K5 growing too fast?
o Yes 18%
o No 23%
o Doesn't matter 36%
o Inoshiro 20%

Votes: 130
Results | Other Polls

Related Links
o Slashdot
o Scoop
o Also by Signal 11


Display: Sort:
Voting, the story queue, and thoughts on K5 | 49 comments (38 topical, 11 editorial, 0 hidden)
my thoughts (3.66 / 6) (#5)
by Defect on Fri Jan 26, 2001 at 12:38:16 PM EST

I honestly think scoop is a little oddwards in it's thinking. You need a login to do much of anything on k5, post, submit, make a diary, vote, etc, but not everyone does all of those things, and some may not vote. Scoop should take into account the percentage of users who vote rather than the hard number of how many users there are.

Check out my latest diary for my thoughts (which was written before the sudden outbreak of meta articles in the queue on this topic). Anything else i say would already have been covered there.
defect - jso - joseth || a link
Prefer your idea (4.66 / 3) (#8)
by ucblockhead on Fri Jan 26, 2001 at 12:46:08 PM EST

I prefer your idea of tracking "active" users. The danger of hard thresholds is that if the number of users got high enough, it would become easier for a small minority to trash a story before the majority of interested parties got to see it.

Perhaps it also might make sense to track how often votes are coming in. If an article hasn't got voted on in a day, seems to me that it is time to dump it regardless of its score. Either that, or perhaps add 10 points to the "dump it" threshold every day.
-----------------------
This is k5. We're all tools - duxup
[ Parent ]
I liked my idea better (4.00 / 8) (#6)
by delmoi on Fri Jan 26, 2001 at 12:44:00 PM EST

When you first view a new story in the queue, kuro5hin would assign you, say, a 75% chance of being able to view it.

I don't see how having less people be able to vote on something would make it get posted quicker

There are some other related changes I've been mulling over - one of which is removing topical comments from stories while they're in the queue

It used to be this way, but everyone was posting long, thought out comments anyway

Anyway, I liked my idea better. Go by a sliding ratio. As a story gets more votes, the ratio required to post goes down as well.
--
"'argumentation' is not a word, idiot." -- thelizman
I can't sympathize (3.87 / 8) (#7)
by spaceghoti on Fri Jan 26, 2001 at 12:44:20 PM EST

This is a logical evolution of the website that we all (or at least most of us) saw coming. I don't see that complaining about it is going to serve any constructive purpose. We either adapt to the changes in our environment or we go looking for someplace else we like better. Myself, I plan to adapt.

If this site becomes like that other site, so be it. Nothing we do is going to stop it. I can't say I'm happy about that thought, but I'm not inclined to try to impose artificial restrictions on the growth or evolution of K5. It's kind of like legislating sex; you can pass all the laws you want about it but it's still going to happen, and in ways you may not necessarily approve of. If anything, it'll get worse because you're trying to restrict it. People aren't very good about arbitrary decisions involving behavior.

This is not meant to be a flame. I mod up and vote up a lot of your work. I just happen to disagree with you this time.



"Humor. It is a difficult concept. It is not logical." -Saavik, ST: Wrath of Khan

Ideas to reduce queue size. (4.71 / 7) (#10)
by Mantrid on Fri Jan 26, 2001 at 01:46:11 PM EST

I think that there should be some way for a story to go stale. If it's been around for a day or more and hasn't been approved, perhaps it should be deleted. Or maybe it could start receivng auto-generated negative votes at some specified interval. I'm not sure about the specifics, but if a story has been hovering for a couple of days the original author may not even care anymore.

Also perhaps an author should be able to delete his/her own post under certain conditions...I had one in there that I realized after (due to editorial comments and rereading), probably wasn't that good a story. I think it still might be sitting there dying a slow and painful death...when really it should've been put down humanely yesterday hehe.

I don't think I like the 75% idea at all myself. Restricting comments to being editorial until the story is approved might be good though.

I can see where you're coming from though...there's been quite a few and after voting for a bunch of posts I stopped thinking things through quite as much.

Also I was wondering if removing the 'i don't care' option might be a good thing...I dunno. Would 'not caring' infer that you don't want to see it on the front page?

good idea (5.00 / 1) (#11)
by dze27 on Fri Jan 26, 2001 at 01:50:42 PM EST

I like the automatic negative votes, how about a -1 every hour? That would stop stories from hanging around more than 2 or 3 days in most cases.

And I agree with the poster, everyone should see every article as it's submitted, that's the whole point of the story posting procedure.


"Luck is the residue of design" -- Branch Rickey


[ Parent ]
Hourly bitrot (none / 0) (#33)
by kmself on Fri Jan 26, 2001 at 07:21:56 PM EST

I like this in general, though:

  • K5's usage stats are much higher during (US) daylight than nighttime hours. Not sure what the time standard is, believe it's US Eastern. Peak use is ~1-3 pm. This means that stories posted at (sever-relative) night will tend to start disadvantaged.
  • I'd like to see an accellerated degredation but a lag prior to application. Maybe lose 1 point an hour for the first twelve hours of life, then double the bitrot for each additional six hours of life. Hmmm: 1-12: 1, 13-18: 2, 19-24: 4, 25-32: 8, 33-38: 16.... Shouldn't see too many stories over a day or so old at that rate. Modulo values -- I think that decay may be too high, maybe make it every 12 hours instead.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Expiring stories (4.85 / 7) (#12)
by rusty on Fri Jan 26, 2001 at 02:16:08 PM EST

I actually like how Drop.org's code, Drupal, expires stories. Basically, you set an "expire" threshold, and when a story has that many votes, without being posted or dumped, it goes away. So I could say "If 10% of users have voted, and we still haven't reached a concensus, drop it." If no one has any major objections, I was going to add this.

____
Not the real rusty
I have another idea (4.20 / 5) (#14)
by gblues on Fri Jan 26, 2001 at 02:47:07 PM EST

How about having the system automatically give stories a -10 after 24 hours, and an additional -10 every 12 hours afterwards until it hits the "dump" threshhold?

I've noticed that stories that tend to get posted generally do so within a day of being posted (under normal circumstances, anyway). A side-effect to more stories being posted is that only the really good ones will get enough votes to be posted. When the queue backs up, the tendency is to vote "I don't care" on most of the entries (I know I'm not the only one who does this), and as a result the queue remains clogged. The automatic -10 would essentially flush dead stories out of the queue within a few days.

A "Vote all [0,1,FP,-1]" feature would be nice, too.

Nathan
... although in retrospect, having sex to the news was probably doomed to fail from the get-go. --squinky
[ Parent ]
Maybe...? (none / 0) (#21)
by Khedak on Fri Jan 26, 2001 at 04:19:32 PM EST

Perhaps at the time that threshhold ends, rather than dumping it based on the regular consensus, you could set a more lax consensus? I don't think the problem is with too many stories being accepted, it's just with too many in the queue.

Unless you wanted to keep the pace of the stories slow, like before.

[ Parent ]
Post controversial stories to section? (4.00 / 3) (#23)
by caadams on Fri Jan 26, 2001 at 04:48:20 PM EST

If a story has at least a few hundred people who think it is worth discussing, and the negative voters have not been able to persuade enough people to dump it, why should the story be dumped? Obviously the story was popular enough to get a large number of positive votes, but not popular enough to be worthy of the front page.

My concern is that controversial stories will be dumped more frequently under such a system. The voting system already requires a supermajority of voters to post a story--why make it even easier to dump stories?

I think the current voting queue has already made a deep division between queue-commenters and the rest of the K5 community. Several of the current queued stories have large amounts of discussion that the non-queue readers will never see (since the stories will be admin-dumped after a week or so). I think that dumping even more stories will deepen that division. People who vote +1 may start to feel that they should comment in the queue, because otherwise they won't be able to effectively discuss the story (when it's dumped).

Then again, my vision of the ideal "online community" is more like a library or conference than a single unified "mass media" model. I would rather focus on letting cooperative groups work together than trying to make a single list of approved content. I realize my ideas may not be "the right thing" for K5, but I hope you'll reconsider the 10% vote -> dump feature.

[ Parent ]

Number of topical comments (4.00 / 1) (#25)
by ContinuousPark on Fri Jan 26, 2001 at 04:59:26 PM EST

I had a story in the queue for a week or so until it finally got posted. But it had very few comments attached to it. So I thought, well, it's obvious that, even if it get posted, it's not interesting to the K5 community, as indicated by the small number of comments.

So I agree with your idea of the "expire threshold" and it should be applied to stories like mine but maybe it shouldn't be activated if the story has a lot of (topical) comments as this would indicate that it's indeed interesting at least to a significant part of the community and it should be given the chance to see if it gets posted eventually.

Or maybe you could also consider the ratings of the comments; they could be rather high even if there's just a few of them. In some cases, I think these quality posts shouldn't go down with the story just because it expired.

[ Parent ]
That's pretty good (4.50 / 2) (#28)
by rusty on Fri Jan 26, 2001 at 05:59:38 PM EST

Maybe instead of being a hard "dump" decision, the "stop voting" threshold should make it's own decision based on other factors. Score would play a part (a story that's at 89 when the threshold is reached is obviously better-liked than one that's at -2), as should number of comments, probably average comment rating (possibly weighted by number of raters for each comment, so that a 3.5 with 20 raters weighs more than a 5.00 with one). I think that could be a good compromise between totally algorithmic posting and rigid user-driven posting. Plus, it would be easy to implement. I like this. Any suggestions on what factors would weigh in and how much they should count would be appreciated.

____
Not the real rusty
[ Parent ]
Number of topical comments doesn't mean it's good. (none / 0) (#49)
by John Milton on Tue May 01, 2001 at 12:27:49 PM EST

Some stories generate a lot of comments on how much everyone hates them and how stupid the author is. I would agree however if the story is rated over say 80.


"When we consider that woman are treated as property, it is degrading to women that we should Treat our children as property to be disposed of as we see fit." -Elizabeth Cady Stanton


[ Parent ]
On implementation (4.33 / 3) (#13)
by ContinuousPark on Fri Jan 26, 2001 at 02:29:15 PM EST

Another idea I'd like to bring to the table is to formalize a way of getting regular interviews (round tables) on Kuro5hin. Perhaps taking the top stories of the week and then posting a followup from experts in those areas and doing some Q&A. Kuro5hin has a focus on culture instead of just news, and I would like to keep moving in that direction by revisiting top stories in some fashion.
I find this interesting. But who would conduct these interviews and in who's behalf. One limitation (if you could call it a limitation) with sites like K5 is that, while some members of the community provide very well written and researched content, they aren't in the K5 staff. So they can't go to a press conference or ask for an expert comment saying "I'm from K5, I'm writing an article in X, I would like to ask you some questions".

Now, you could do it in a freelance way but it could be easier if you could say that you're a K5 writer instead of just a K5 poster. But I don't see how this could be implemented, Rusty would have to pick them among people he knows and trusts or maybe a new category of "super trusted" users could be created but this leads to professional ethics considerations.

Another, more viable idea is that somebody proposes a story, the things to be considered (pros and cons, for instance) and then some people sign up , formally or informally, to do some research for the story or you could invite just certain users you trust or like. The person who proposed the idea would become its editor; maybe at the end, the people involved would have to vote for approval. We would have then collective stories that are very well researched, with several points of view integrated in them. Any ideas as to how any of these ideas could be implemented?

One user, one vote (4.62 / 8) (#15)
by Inoshiro on Fri Jan 26, 2001 at 02:52:15 PM EST

Everyone deserves to see every submission that is not voted one way or another. With our growth, we've seen a lot more marginal submissions. You complain of these as they moulder away in the queue, and I agree that the boundry situations need to be adjusted so the crap will sink faster (and the good stories will float above the crap).

I do not agree with hiding stories in the queue from voters. If they don't want to vote, they can skip voting on some stories. We're growing to the point where we can specialize like that. Hopefully we'll be able to make it faster people to vote on stories, so it doesn't take as much time.

The inevitable solution is, of course, to split the site more along the lines of the sections, with each section having its own queue, and each user deciding which queues they want to work with on a regular basis via user prefs.



--
[ イノシロ ]
Agreed, but... (4.33 / 3) (#18)
by Signal 11 on Fri Jan 26, 2001 at 03:24:11 PM EST

I agree completely, however I felt that if I had an idea to share, I would share it, even if I disagreed with it. Afterall, K5 is a community now, and even bad ideas need to be voiced, if for no other reason than to remind us there are less optimal alternatives! Someone has to fall on the sword. :)


--
Society needs therapy. It's having
trouble accepting itself.
[ Parent ]
Wait a second... (none / 0) (#29)
by Luke Francl on Fri Jan 26, 2001 at 06:06:46 PM EST

You disagree with your own idea, but then post it anyway, without saying anything about it?

You are a strange one.

[ Parent ]

Strange? (4.00 / 1) (#36)
by Signal 11 on Fri Jan 26, 2001 at 07:30:03 PM EST

You are a strange one.

Indeed.


--
Society needs therapy. It's having
trouble accepting itself.
[ Parent ]

Not really (5.00 / 1) (#37)
by Inoshiro on Fri Jan 26, 2001 at 09:04:26 PM EST

Ever heard of brainstorming? It's all about throwing out as many ideas as you can, even if you don't personally agree with them.



--
[ イノシロ ]
[ Parent ]
Sorry, Signal 11, but I disagree (4.60 / 5) (#16)
by xah on Fri Jan 26, 2001 at 03:20:32 PM EST

While I agree that something has to be done, everybody should still have access to all of the submissions.

An alternative proposal: if the number of submissions in the queue reaches a threshold (say, 15), then an additional method of deleting posts comes into effect. Every submission over X days old gets deleted, where X is a value calculated with a number of variables inteded to produce a result where X is relatively high.

A second alternative. If the threshold is reached, the impact would be to automatically delete posts that do not have a Positive posting percentage greater than X %, where X is 35 or 40, or so. This would have the impact of making "I don't care votes" equivalent to a "light 'dump it'."

Please feel free to rip off these ideas if you like.

Interesting (none / 0) (#24)
by regeya on Fri Jan 26, 2001 at 04:50:29 PM EST

I was thinking the same thing a while back, or at least something similar. I was thinking that perhaps a story could go through a process of natural decay. It's easy enough to implement, or at least I think it would. I haven't touched the scoop code so I don't know what the difficulty would be.

[ yokelpunk | kuro5hin diary ]
[ Parent ]

Degrading scores over time (4.25 / 4) (#19)
by tdowney on Fri Jan 26, 2001 at 03:28:50 PM EST

In one of the previous iterations of this post someone had the idea of degrading story scores over time. It sounds like a good idea to me. For example, every hour the story score would drop by 5. This would accomodate the drop off in momentum that seems to be occurring with many of the stories languishing in the queue.

How does that help? (4.00 / 1) (#20)
by Khedak on Fri Jan 26, 2001 at 04:14:12 PM EST

In one of the previous iterations of this post someone had the idea of degrading story scores over time. It sounds like a good idea to me. For example, every hour the story score would drop by 5. This would accomodate the drop off in momentum that seems to be occurring with many of the stories languishing in the queue.

I don't see that you've considered this, but what if the story score is currently increasing at a net 5 votes per hour? Wouldn't your measure simply cause a story that would otherwise be posted in a couple of days to stay in the queue indefinitely, in that case?

I think the solution is simpler: Put a time limit on the stories in the queue. At the end of the limit (3 days perhaps?), it's posted or dumped depending on which score (post or dump) is higher. Don't care votes are not counted and considered abstentions.

[ Parent ]
That could have interesting side effects (4.00 / 1) (#26)
by aphrael on Fri Jan 26, 2001 at 05:34:15 PM EST

involving bottlenecks.

Having hourly time-based reductions implies that a story is more likely to successfully post if originally submitted at a time of day with high readership, and unlikely to successfully post if originally submitted at a time with low readership (imagining an absurd example, 4 hours with nobody rating a post would automagically consign it to the dustbin).

People will figure this out by experiment and learn to time their posts correctly --- causing an avalanche of new posts within a small time block.

[ Parent ]

Preference? (4.83 / 6) (#22)
by Khedak on Fri Jan 26, 2001 at 04:25:15 PM EST

Maybe, rather than dump it randomly, a user could define what subset of sections they wanted to read and moderate. For example, if I don't want to read MLP, then I don't see MLP in the front page nor do I have to moderate it. I have a strong tendency to vote "don't care" on most of the MLP's I see, but I feel a need to contribute to the vote flow in the queue, it being so backed up and all...

Of course, the percentage calculation will then have to be adjusted to not be based on the total number of users, but on the number of users who have decided to read and moderate that section.

Very good idea (none / 0) (#35)
by kmself on Fri Jan 26, 2001 at 07:28:35 PM EST

This is a suggestion I've been leaning toward for quite some time. Plain and simple -- I don't care what goes on in most of the "Culture" and "Politics" posts. I would prefer many of the people who appear to be voting down Tech, Meta, and Internet posts didn't. Story moderation by section makes IMO much sense.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Buh... (3.50 / 4) (#27)
by J'raxis on Fri Jan 26, 2001 at 05:39:39 PM EST

When you first view a new story in the queue, kuro5hin would assign you, say, a 75% chance of being able to view it. If you can view it, you can vote on it. Otherwise, it's hidden from you until it is posted into the database. This way, we can each moderate maybe 5 submissions each day on average. We could, of course, always see our own stories.

Bah. Only letting people vote on a random selection of stories? This sounds like trying to mutate that site's metamoderation system into working on stories, not just comments.

There are some other related changes I've been mulling over - one of which is removing topical comments from stories while they're in the queue. Before you jump to conclusions, I think it is necessary for several reasons - often a story is not posted in which posters spend a lot of time composing a good reply.

Now this I completely agree with. I've avoided posting comments several times because it looks like the story is sinking, and then I often end up forgetting to go back and post them if the story somehow makes it up.

Kuro5hin is growing, and with it is a crush of new users and new stories being submitted into the queue.

Of the 25 stories I see in the queue, four of them are yours. ;)

-- The Raxis

[ J’raxis·Com | Liberty in your lifetime ]

Length of queue (4.00 / 2) (#32)
by enterfornone on Fri Jan 26, 2001 at 07:15:58 PM EST

Why not just dump stories that are still in the queue after one or two days. Perhaps drop the posting threshold a little to make up for it.

Another solution would be to decide to post/dump based on the majority vote (or if 60% vote it up or whatever) after a certain number of votes.

Not sure if I like the idea of no topical posts. I can see the reasoning (too many things are discussed to death while in the queue then die once they hit section). But it would just encourage people to post topical comments as editorial.

In short, the problem isn't how long the queue is, it's that it can take a week to decide whether a story is worthy of being posted, after which the story is dated and already discussed to death. We need a way of getting stories out of the queue quickly, while still giving everyone (not just the reloaders and cabal) a chance to vote.

--
efn 26/m/syd
Will sponsor new accounts for porn.
I was pondering this myself (4.33 / 3) (#34)
by Osiris on Fri Jan 26, 2001 at 07:24:06 PM EST

And actually, I thought about posting a story about it last night, but had a midterm to study for :). I don't think the increased length of the story queue is due to the increased threshold to post. After all, the threshold is related to the number of users, right? So the same proportion should be able to post it just as rapidly as they used to.

I think what we are seeing is a function of diversity. As the population size increases, there is no longer so much of a consensus as to what a good story is, resulting in a larger trend towards 0 as the score for any story. I have no idea about accurate site statistics like # of voters, stories posted/day, or whatever, either now or over time, but a statistical study might be interesting. I mean, you might expect that as the number of visitors grows, the #stories/day would go up, and it has, in the queue, but looking back over the history of the site, the number of stories posted/day has remained fairly constant, or has grown much slower than the # submitted/day.

Anyway, to fix it, I have a suggestion (you saw that coming, right?). Rusty mentioned having stories drop out after a certain number of votes, if they haven't been posted by then. I'd suggest the following addenda:

  1. Have a positive threshold, like now, which means "instant post", and a negative which means "instant drop". Hit either of these, and you're out of the queue one way or the other
  2. If a story doesn't hit those, let it stay for a given number of votes equal to some portion of the registered users. I was originally thinking days, but %users sounds more elegant. Rusty suggested 10%, sounds great for a trial, might need to be adjusted one way or the other if stories don't hit it very fast (a day or two)
  3. After this number is reached, look at the rating. If it's negative, dump it. If positive, section it. If zero, um, do a shot, I dunno.

Eratta, comments, algorithmic differences, help on EE homework?



K5 Story Queue (4.71 / 7) (#38)
by Wil Mahan on Sat Jan 27, 2001 at 12:19:57 AM EST

Given the techno-centric nature of this site, I'm surprised that in the design of the the current system, as well as in the proposals to fix it, I haven't seen many ideas based on statistical methods. Of course, I could be wrong in my criticism; I don't claim to be anything close to an expert. The following is an idea based upon but an elementary understanding of the practice of statistics. Please correct any of the mistakes which I have doubtless committed.

The Problem
The basic flaw with the current system is that it doesn't seem to be well thought out from a statistical point of view. The concept of thresholds being a percentage of the total number of users completely ignores the Law of Large Numbers. Many people seem to not realize that a sufficiently large random sample of users could represent the "average" opinion of all K5 users, to an arbitrary confidence level. So even if K5 had a million users, votes from a random several hundred would be quite an accurate indicator of the general opinion! Thus I think there is no reason to use any kind of increasing thresholds (and good reasons not to).

Something else I don't understand is that the current system doesn't take into account statistical trends. Suppose rusty[1] posts a story that everyone can agree belongs on K5. If 14 out of the first 15 voters want the story posted, then we can say with 99% confidence that more than 75% all K5 users want the story posted. But the system naively waits for the number of votes needed to reach a threshold, which is especially bad for time-sensitive stories, such as news.

My final complaint before I propose a solution is that there are different post and dump thresholds. If a story's score rises at a certain positive average rate over time, it will likely be posted; won't increasing the threshold above a certain amount just delay the inevitable? The reverse seems to occur for small dump thresholds--the stories that are dumped quickly with small negative thresholds would probably have been voted down anyway. N.B. I could be wrong on this point, though.

There are a few other, more minor problems, but I think you get the idea.

A Solution(?)
So it's easy to spout criticisms of a working system, especially when some of them might not even be valid. What can be done to fix it? (Sorry about the lack of symbols in what follows; it's difficult to do math in HTML. :-( ).

Suppose that for a given story, there is a certain proportion of all K5 users who want the story posted (# of supporters) / (# of users eligible to vote); call it Pi. So a proportion of 1-Pi don't want it posted. (From here forward, the votes of users who "don't care" will simply be ignored for the purposes of scoring posts, which seems to be the most natural choice.) Because of an unfortunate disparity between the languages of K5 and statistics, the "post" votes will have a value of one, while "dump" votes are represented by zero.[2]

It seems reasonable to post a story if and only if at least a certain proportion Pi_0 of the K5 voting population wants it posted (i.e. Pi > Pi_0). Of course, that proportion could be easily adjusted (perhaps on a per-user basis?!) to affect the quality of the stories posted. Let us also assume that people who vote represent a random sample and that votes are not correlated with the time at which they are submitted (not really true, but I believe close enough to the truth). Say n people vote, and call the proportion of voters who want the story submitted (i.e. (# post)/n) p_hat. Then it is relatively simple to use p_hat to test the null hypothesis that Pi is greater than Pi_0. If the null hypothesis is accepted, then the story is posted; otherwise, the hypothesis that Pi < Pi_0 is tested; if that hypothesis is accepted, the story is rejected (confused yet? :) ); otherwise, voting continues.[3]

Glossing over much of the statistics, the Central Limit Theorem tells us that p_hat is distributed normally[4], and it is not difficult to show that the average of the numerical scores (remember, post=1 and dump=0) is p_hat and the standard deviation is sqrt(Pi*(1-Pi)/n), which for deep statistical reasons we approximate by sigma=sqrt(p_hat*(1-p_hat)/n). Then the z-score is (Pi_0-p_hat)/sqrt(p_hat*(1-p_hat)/n). The z-score represents a value on the standard normal curve, and an area under the curve represents a probability. We can thus ensure that our method is accurate 95% of the time by comparing the z-score with z_star=1.96. Perhaps better would be 99% confidence, which corresponds to a z_star of 2.58. The drawback to a higher confidence level is that more votes are required to reach that confidence. If the z-score is greater than z_star, the story is posted; otherwise the process continues as described previously.

Well, sorry about the long post, but this is just an overview of my thoughts about the K5 submission system. I can think of a number of problems with a system like one that I've described, but I think I've written enough for now and others can probably think of better objections anyway. Any comments on whether such a system would be feasible would be especially appreciated. While the math and statistics may have errors, those can be fixed; problems with the assumptions that the system makes and applying it to K5 could be more challenging.

Regards,
Wil Mahan

Notes
[1]Or anyone else, but this seems to happen to him most frequently. :) Conversely and perhaps more importantly, this criticism also applies to spam in the queue, unless an admin deletes it.

[2]This convention might or might not be reflected in what the voter actually sees. There wouldn't be any need to change the 1/0/-1 convention except in the code, but I am attracted to the "all posts are assumed to contribute" aspect of a 1/0/"don't care" system.

[3]Note that once the required precision is reached, the story will be posted or dropped. So if we are "certain" that 55% of the users want it posted, but the threshold is 60%, then the story is dropped. That might seem obvious, but it is different from the current system in that the same criterion is used for both accepting and rejecting.

[4]The method I present here uses the Wald normal approximation to the binomial distribution, which is considered valid when min(np,n(1-p))>5. I would have preferred to use the more exact Clopper and Pearson method, which is based on the binomial distribution, but I must confess that I can't remember a few of the details of that method and I don't have a statistics book handy. :-( In practice, the results should be virtually identical regardless of which method is used.



Not always (none / 0) (#39)
by DeadBaby on Sat Jan 27, 2001 at 12:47:28 AM EST

Of course lets say the story is posted at 2am. Your sample group could end up not reflecting the actual readership very well. (given the number of people who are awake and willing to vote)
"Our planet is a lonely speck in the great enveloping cosmic dark. In our obscurity -- in all this vastness -- there is no hint that help will come from elsewhere to save us from ourselves. It is up to us." - Carl Sagan
[ Parent ]
timezones.... (none / 0) (#46)
by thomas on Sun Jan 28, 2001 at 07:07:31 PM EST

of course, 2am in one spot on the planet is 2pm in another.

War never determines who is right; only who is left.
[ Parent ]

I really disagree with you on most of this (4.00 / 1) (#40)
by jesterzog on Sat Jan 27, 2001 at 02:13:25 AM EST

My solution is to set the thresholds to a static amount, and instead vary the percentage of the total users who see the story in the queue. When you first view a new story in the queue, kuro5hin would assign you, say, a 75% chance of being able to view it.

It sounds like you're trying to make it too complicated, and that's somewhere that slashdot (IMHO) has fallen over. When the code's in place and this doesn't work, would we just throw even more complicated systems on top of it?

There are some other related changes I've been mulling over - one of which is removing topical comments from stories while they're in the queue.

This has been proposed several times in the past and all I can think is No. If I can't post a topical comment when a story's in the mod queue, I have to click on and read that story twice! That's twice as many mouse clicks. You can bet that if you do this, people will start posting topical comments as editorial, anyway.

My usual routine is to check the everything page for anything i've missed, then go to the mod queue and run through all the stories there. The next day I'd often have trouble remembering if a story was posted the last time I read it, or not, and if anything that's only going to cut down on the amount of potentially insightful posts that people write when their brain's warmed up to a new idea.

often a story is not posted in which posters spend a lot of time composing a good reply. I don't think it is fair to them to have their comments "eaten" by the system

I think most people who can tell the difference between editorial and topical comments are capable of making a judgement call about whether a story is likely to be posted or not. If you have something long to say but don't want to risk wasting your time, there's a simple solution. Don't say it until the story gets posted. Let people comment topically on a story in the mod queue at their own risk, but don't deny people their ability to comment.

Kuro5hin articles often are related to previous Kuro5hin articles. Often it is a different approach, but the context is the same. What I would like to see is a more intelligent "What's Related?" linking those other articles.

I'm inclined to vote a lot of these stories down when they come up because they've already been covered - unless there's a good reason not to. People should do some research before posting a story about what's already been covered, and if it's related they can post their own links. Why do we need a computer to try and work it out?

I don't think it would need to be automated as you say. IIf we had some more fields for people to enter related links into, it'd be difficult to stop people using them - although it would likely be for the wrong purpose. But how many should there be, how often would they be used properly, and would people feel compelled to start adding rubbish and unrelated joke types of links to them as we're getting in a lot of polls?


jesterzog Fight the light


How about this... (4.33 / 3) (#41)
by finial on Sat Jan 27, 2001 at 10:47:26 AM EST

How about if you can't post a "topical" comment until the post is voted in? Only "editorial" comments would be allowed for messages in the queue. It seems to me that by the time a posting reaches either the front or section page, it's already got most of the posts it's going to get, I've already read them and have no need to read it/them again. Look at this thread. It's been here for one day and has 40 responses and it's not even posted (to a page) yet.

This would do several things:

  • It may not make the "bad" ones drop off any faster, but it would speed the "good" ones through the queue so that people could respond.
  • It would slow down the responders a bit so that they would have a chance to, um, consider what they are saying and get their thoughts in order. If you have an urgent need to comment on something that has not yet reached publication, put it in a diary and wait.
  • Since only editorial comments would be allowed for postings in queue, all of the "This is news not MLP," "This is MLP not news," "This isn't even MLP" complaints could be taken care of before it goes "live."

One of the risks is that people may start posting topic comments as editorial comments, but no system is prefect.



That is simply a bad idea (4.40 / 5) (#42)
by ribone on Sat Jan 27, 2001 at 11:06:10 AM EST

My solution is to set the thresholds to a static amount, and instead vary the percentage of the total users who see the story in the queue. When you first view a new story in the queue, kuro5hin would assign you, say, a 75% chance of being able to view it. If you can view it, you can vote on it. Otherwise, it's hidden from you until it is posted into the database. This way, we can each moderate maybe 5 submissions each day on average. We could, of course, always see our own stories.

I don't agree. Hiding the stories from 75% of the users while it is in the queue is bad. Everyone should be able to vote on a story's worthiness. That's what K5 is all about: voting on what stories get posted. If you can't do it a better way without modifying it in such a manner, then you need to leave it alone.

I don't know all the details about how scoop works, but I don't see much of a problem right now. Yes, stories have been in the queue longer (much longer, in some cases, than necessary), but I think there has to be a better way to solve the problem. Personally, I'd rather see all the stories and get to vote on them and have to wade through a bit more stuff in the queue than go with what you're suggesting.

Notice that I'm not offering any solutions here, I'm only rejecting yours.



Should have been two separate submissions (none / 0) (#47)
by Tim_F on Mon Jan 29, 2001 at 12:37:54 AM EST

Not a bad post, Signal 11. But you should have split it up somewhat. A separate story about your ideas regarding linking and discussions where nice.

How to make ME vote more.... (none / 0) (#48)
by reaper20 on Sat Mar 10, 2001 at 01:05:53 PM EST

I think the queue would work better if more people voted (duh). I think the format of the story queue is why people don't vote more. If the queue page was formated to I can scan and vote on ALL THE ARTICLES at once, then it would work better. Make it the same format as a comment page, showing enough of the article to form an opinion, maybe with a 'read more' link. That way I can vote once or twice a day without having to reload eavry stinking time I want to vote. That gets annoying, especially on dial-up...

Voting, the story queue, and thoughts on K5 | 49 comments (38 topical, 11 editorial, 0 hidden)
Display: Sort:

kuro5hin.org

[XML]
All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!