Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

[P]
Story Queue Changes

By rusty in Meta
Sat Jan 27, 2001 at 11:28:57 AM EST
Tags: Kuro5hin.org (all tags)
Kuro5hin.org

There's been some clamor lately that there are too many stories hanging around for too long in the queue. I think we've finally reached the point where there's enough submissions and enough voters that straight additive voting alone doesn't quite cut it. So I've made some changes to the voting system, as described below. Hopefully this will help out with stories that don't have a clear voting majority one way or the other, while still keeping what was good about the existing voting system.

Update [2001-1-27 14:39:28 by rusty]: Some of you may have noticed that there was a pretty bad bug which prevented anything from happening, even if the vote threshold was reached. That should be fixed now. If you are the author of a story in the queue, and you got a bunch of identical emails about it, I'm really sorry. That shouldn't happen anymore. Nothing like alpha-testing on the live server, eh? :-)


ADVERTISEMENT
Sponsor: rusty
This space intentionally left blank
...because it's waiting for your ad. So why are you still reading this? Come on, get going. Read the story, and then get an ad. Alright stop it. I'm not going to say anything else. Now you're just being silly. STOP LOOKING AT ME! I'm done!
comments (24)
active | buy ad
ADVERTISEMENT
The Problem:

Some stories are really good. They fly out of the queue like a bat out of hell, or at least post within a day or so. Others are, well, really bad, and they tend to drop like rocks. This covers probably the top 10% and the bottom 1% of stories, in my guesstimation. But then there's the other 89%, which aren't clearly loved or hated by everyone. Sometimes they're controversial, sometimes they're just not tremendously inspiring. These can hang around for days and days, often racking up very good discussions, but never posting or dropping. Clearly, something needed to be done about this.

The Solution (for now?):

I read on Drop.org that Drupal, their codebase, has a third threshold, a "maximum votes" threshold. Basically, when a story reaches this number of votes, without posting or dropping, it is dropped.

"Hey," I thought, "that's a pretty cool idea." At least, it can tell you when the community is split on something, and it's time for a decision to be made. So the first change is that in additon to the normal post and hide thresholds, there's now a "maximum votes" one as well. Right now, it's set at 6% of users, or about 720 votes. It seems that the stories in the queue longest have around that many votes. We'll see how it works out, and raise or lower that number as needed.

But that wasn't totally right, I thought. I didn't like the idea that we just drop stuff when it's been around for a while. Maybe there's a really good discussion going on, that would make it worth posting the story, even if it wasn't in itself a pearl from heaven.

So I decided we should run some heuristics before deciding what to do with the story. They ought to take into account the votes that have been cast, and the comments that have been posted. So I decided comment ratings were a good thing to key on, and that we could work out a decent idea of where the story stands as follows:

  1. The Voting Score:

    Assign each vote a score, keyed to the comment rating scale. The scores are as follows:

    • +1, Front page: 5
    • +1 Section: 4
    • 0 Don't Care: 2
    • -1 Dump it: 1
    So, "Don't care" votes are treated as slightly negative. I'm guessing that apathy isn't really a positive feeling about a story, so there's a slight "apathy penalty" for stories people really didn't care much about either way.

    To get the "voting score" of a story, then, I just do:

    (FP * 5) + (Sec * 4) + (DC * 2) + (Dump * 1) / Total Votes
    This will give us a number between 1 and 5, representing a rough idea of the overall voting picture.

  2. The Comment Score:

    Then, take the ratings from all the topical comments in the story, and figure out a weighted "comment average". The weighting is done by number of ratings, so that, for example, a comment with 17 ratings that has a score of 3.5 counts like 17 comments at 3.5. A comment with one rating of 5 will count like one comment rated 5. This is part of the weighting used in Mojo, and the idea is that it favors ratings that have a higher number of raters, and thus more "confidence" that the rating is pretty close to what the comments rating ought to be.

    Editorial comments aren't counted in this, because in theory, you're deciding whether the topical conversation has been interesting enough to sway our decision about what to do with the story, and editorial comments aren't normally shown once a story is posted anyway.

  3. The Final Score:

    Finally, we average those two scores together, to get a final score for the story. This will also be between 1 and 5. If it's above a certain value, the story will be posted to section-only. If it's above a higher value, the story will go on the front page. Right now, those values are 3.25 and 4, but these are basically educated guesses. I have it set up to email me when a story is posted by the system this way, with data I can use to decide if these scores are any good or not. They can be adjusted, and will be if necessary.

So the upshot of this is that there is now at least a firm endpoint for voting. I don't have to make any more of those horrible decisions, where I drop a story that's been around forever, despite it having some great threads going on. And you shouldn't have to watch the same stories linger in the queue for ages.

This system probably favors posting over not posting, slightly. Normally, comment rating averages are above three, and if a story has >700 votes and isn't posted yet, the voting score is very likely going to come out just slightly on one side of three or the other as well. I think this is probably a good thing. I think we should be keeping more stories, because even average to middling stories are known to spawn very good discussions, and that's really what we're here for.

I'd like everyone's thoughts on this. Ideas? Suggestions? If something strikes you as very bad about this scheme, please say so! If you have a "feeling" that this might be bad, I urge you to wait a couple weeks and see how it goes before proclaiming the already over-hyped "Death of K5". As always, this is an experiment. Let's see if it works!

Sponsors

Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure

Login

Poll
Post-Bot:
o Good! 17%
o We'll have to see. 26%
o Bad! (and I explained why in my comment) 2%
o Good God, man, you've given the Machine sentience! 27%
o You're obsessed with averages, Rusty 19%
o Driph must die for crimes against humanity 7%

Votes: 157
Results | Other Polls

Related Links
o Drop.org
o Also by rusty


Display: Sort:
Story Queue Changes | 63 comments (63 topical, editorial, 0 hidden)
Mostly a good idea (3.84 / 13) (#1)
by aphrael on Sat Jan 27, 2001 at 04:41:17 AM EST

but it might need some tweaking. In particular, i'm worried about the effect that zero-rated stories could have on a thread's comment rating; IE., if someone decides to spam and/or troll a thread, and lots of people rate the posts at zero (i've seen some things with 35-40 zero ratings in the hidden queue), could that be enough to cause a thread to tank? If so, then maybe zero-rated stories should be ignored for that purpose.

Also, it isn't clear from your post, but can I assume that up to the drop-dead voting limit is hit, stories will post under the old algorithm? IE., if there are only 200 votes on a story, the old rules apply?

Yes. (4.50 / 4) (#5)
by rusty on Sat Jan 27, 2001 at 05:01:44 AM EST

Dammit. I think I forgot to exclude them, but I meant to prevent comments rated <1 from affecting the comment score. Thank you for reminding me.

And yes, at all points until the voting limit is reached, the system behaves exactly like it used to. It worked well for things that hit a limit quickly-- this change is only meant to affect the ones that don't.

____
Not the real rusty
[ Parent ]

Apathy penalty (3.84 / 13) (#2)
by enterfornone on Sat Jan 27, 2001 at 04:47:13 AM EST

The main reason I vote don't care is because it's an article I'm not interested in and don't really understand (like some articles on programming algorithms and the like) so I can't really tell if it's a good article or not.

So I've vote don't care as a way of saying "let the people who know this stuff decide".

So I don't think there should be an apathy penalty.

Other than that it's great.

Slightly off topic, on front pages. I assume front pages are still based on front votes > section votes. I think it should be front > (section + dump), there have been a lot of stories that make it to the front despite many voting against it, and often I am tempted to +1 section a story I would rather dump simply because I know there are lots of people voting it up.

+1 section BTW :)

--
efn 26/m/syd
Will sponsor new accounts for porn.
don't care and front page (4.60 / 5) (#4)
by rusty on Sat Jan 27, 2001 at 04:59:46 AM EST

The main reason I vote don't care is because it's an article I'm not interested in and don't really understand

Right, so if there are a large number of "don't care" votes, it would seem to mean a large number of people are either uninterested or unknowlegeable (and didn't learn anything from the story) about that. So, I feel that's a slightly negative thing-- you wouldn't want to post a lot of stories people didn't care about, would you?

About front page: If voting is determined in the normal course of things, that is, if a story reaches the post threshold before voting ends, then it's done the old way. If a story is posted by the system, when voting ends, then it's a threshold thing. Getting a aggregate score of 4 with a story that didn't get posted normally is going to be really tough. It's possible, but just barely. I wouldn't expect to see it happen much. I think the existing system is pretty good at finding front-page stories.

____
Not the real rusty
[ Parent ]

The All-Powerful Queue (4.12 / 16) (#3)
by Mr. Excitement on Sat Jan 27, 2001 at 04:58:02 AM EST

Lately, I've noticed that some of the more interesting discussions occur when a story is in the queue.

When hundreds of users come through, vote, and occasionally chime in or rate some comments, you get something of a "timeshare" discussion, since it's easy to stop by, add something, then go about one's merry way. In other words, the entire user base can share the same discussions, but you don't have everybody in the same place at once, unlike a story that gets voted to the front page overnight. I'm not sure exactly how that changes the discussion, but it seems like the queue sometimes acts as its own proto-section.

It'll be interesting to see if the New-and-Improved queue changes this, and whether or not that change is for the better. :)

1 141900 Mr. Excitement-Bar-Hum-Mal-Cha died in The Gnomish Mines on level 10 [max 12]. Killed by a bolt of lightning - [129]

That's right. (4.40 / 5) (#10)
by Seumas on Sat Jan 27, 2001 at 07:08:23 AM EST

Not only do most of the best discussions take place while the stories are in the queues, but often I've completely lost interest in the story by the time it is actually posted out of the queue. Not to say a story is good or bad -- just that after about a day to a day and a half, there's not a lot of discussion added on, unless it's particularly enthralling. And for those of us who watch the queue like hawks, we jump in on things within moments.

All in all, I guess I like Rusty's efforts. I think they're a little complicated and may tend toward eding decent stories out, but the general idea is there -- that being "place a limit on these things!".
--
I just read K5 for the articles.
[ Parent ]

Comment rating average problems (4.35 / 17) (#6)
by Delirium on Sat Jan 27, 2001 at 05:12:43 AM EST

I see three problems with taking into account average comment ratings in deciding whether or not to post a story:

1) Someone who really wants a story to be posted, in addition to voting +1 FP, may go through and rate all the comments to 5 to skew the comment rating average so the story is more likely to be posted.

2) On the other hand, someone who really doesn't want a story to be posted, in addition to voting -1, may go through and rate all posts 0 (or 1) to skew the comment rating average so the story is less likely to be posted.

3) Or the person who really doesn't want the story to be posted could post lots of spam to the article such that other users will rate it 0 and thus (like in situation 2) reduce the comment rating average.

As for how to fix these problems:

Situations 1 and 2 don't really seem fixable; all we can do is hope that the majority of users will rate comments based on the actual comment's merit rather than on their opinion of whether the article the comment is attached to should be posted. If this is the case, it should dampen any comment rating skewing caused by 'malicious' users, though I'm not convinced that the effect will be entirely negligible.

Situation 3 could possibly be fixed by entirely ignoring all comments below a certain threshold for these calculations (say 1.0, so all spam is ignored). This seems reasonable, because spam really shouldn't have an effect on whether a story is posted or not - if it has lots of good on-topic comments, it should be posted, even if there's also lots of spam.

Comment stuff (4.37 / 8) (#7)
by rusty on Sat Jan 27, 2001 at 05:24:26 AM EST

I agree with you that there's little we can do about 1 and 2. Comment rating is like that. We'll see if that seems to happen.

About 3, I already fixed it so that spam (comments rated <1) are not counted. I think that's a pretty safe thing to do.

One thing I do wonder about. A story with "a lot" of comments seems like it ought to have a better shot at being posted than one with not a lot. And yet, chances are, if a story has a lot of comments, people have been visiting it more often, and rating more, and thus the ratings will tend to be *lower* than one with only a few participants. I'd really like to account for this. But how do you determine what "a lot" of comments are?

One idea I have is give a bonus to stories with a large ratio of topical / editorial comments. It seems that stories with more topical comments should be better to post. More editorial comments would seem to mean that people wants changes, or weren't interested in dicussing. Does that make sense?

____
Not the real rusty
[ Parent ]

fun with algorithms (3.66 / 3) (#9)
by Delirium on Sat Jan 27, 2001 at 05:46:11 AM EST

As for 1 and 2, I think this is just the problem that's been around forever on k5 - 'malicious' raters can screw up ratings. The best (only) way to fix this is for lots and lots of people to rate. So I suppose the only change this really makes is that everyone regularly rating comments is even more important than it was before.

And you've already removed problem 3, fast response time considering when you posted your story and when I posted my comment. =P

And yeah the "lots of comments" thing makes sense. One idea would to just give a small bonus per >1.0-rated topical post (regardless of the rating of the post - just give a bonus for it being there). That way a story with lots of discussion will have a better chance of being posted simply because it has more comments. The topical/editorial ratio idea seems like it would have the same sort of effect, except it would combine the "lots of topical posts bonus" with a "lots of editorial posts" penalty. I can somewhat see your justification for this, but I'm not entirely sure if it's a good idea - is lots of editorial comments necessarily bad if there are lots of topical comments as well? Also, if you just calculate a simple ratio, the mathematics leads to screwy results with low levels of editorial comments: a story with a 30:1 ratio would have twice as great a calculation here as a 30:2 ratio, despite, the discussion quality of the two scenarios being roughly the same.

And of course there's always the uncertainty principle: it's impossible to measure the quality of a discussion without effecting that quality through the measurement. If editorial comments represent a penalty (through lowering the topical:editorial ratio) then users who dislike a story might post superfluous editorial comments. These might actually point out minor flaws in the story (and thus not get rated <1.0 and removed from the calculations) but might nonetheless be unnecessary and something that the user wouldn't have otherwise posted; effectively this would add noise to the discussion. On the other hand, users who want a story to be posted might be discouraged from posting any editorial comments whatsoever. This could reduce feedback from people who might have some editorial issue with or comment about the style of the story but who think it's still worth being posted.

So in general I'm a bit wary about the topical:editorial ratio being used, though not entirely opposed to it. I do think just a straight count of topical comments would be a better correction factor.

[ Parent ]

Comment counting (3.33 / 3) (#20)
by rusty on Sat Jan 27, 2001 at 03:40:52 PM EST

The question I have about using the topical comment count is how does it affect the actual score? Like, do we add a tenth of a point for every ten comments, or what? The advantage to using a ratio is it can be bounded. If anyone can come up with a good way to relate number of comments to the 1-5 score, I want to hear it!

____
Not the real rusty
[ Parent ]
Something asymptotic (4.00 / 1) (#50)
by sab39 on Mon Jan 29, 2001 at 12:57:05 PM EST

It sounds like what we need is something that starts at 3 for no posts (or even starts at 2 - it may be fair to slightly penalize an article with no discussion at all, or very little) and approaches 5 asymptotically as the number of posts increase. So how about something like this:

discussion_bonus = 5 - 3 * x^(-n)

where x is a constant chosen such that x^(-N) = 2/3 for some N that's considered a minimum "adequate" level of discussion. Thus if N=3,

n = 0 ==> discussion_bonus = 2
n = 1 ==> discussion_bonus = 2.something
n = 2 ==> discussion_bonus = 2.something
n = 3 ==> discussion_bonus = 3
n = 4 ==> discussion_bonus = 3.something
...
n = 300 ==> discussion_bonus = 4.99something

Then, once discussion_bonus has been calculated, average it into the main score. It may be worth doing a weighted average so that discussion_bonus has less influence than other factors: a story with hundreds of comments will get a discussion_bonus very close to 5, which will tend to overrule other aspects. On the other hand, this could be good, because a long discussion really *is* a very good indicator that something should be posted.

Thoughts?

Stuart.
--
"Forty-two" -- Deep Thought
"Quinze" -- Amélie

[ Parent ]
Moderation abuse (2.33 / 6) (#18)
by Wah on Sat Jan 27, 2001 at 01:17:48 PM EST

I don't know if you've done this already (haven't submitted a story in a while) but I think you would need to remove the author's ability to moderate posts while the story is in the submission que. There's a bit of potential abuse in the system you've outlined. (Signal 11, I'm staring right at you) Sure, a motivated poster could just login with another account, but this might help curb queue abuse.

And I really like the idea of moving "dropped" stories to the poster's diary. It cuts down on the loss of good conversation. Tons of folks have had highly rated, well-thought out comments that are now only available through their user info page and totally out of context. This might increase storage space needs dramatically, but that's what banners are for, eh?

Either way, thanks for trying something. It's better than some webmasters I know.
--
Fail to Obey?
[ Parent ]

why? (4.50 / 2) (#48)
by elby on Mon Jan 29, 2001 at 03:37:51 AM EST

A half-assed attempt at securing a problem such as as described in your first paragraph is never productive. It's a simple matter to make other accounts. In fact, something like that would encourage it.

I'd *rather* the author was moderating under his real account so people could call him on it.

-lb

[ Parent ]

how about a relative measurement of "a lot&qu (4.00 / 1) (#53)
by panck on Mon Jan 29, 2001 at 09:06:40 PM EST

...between all stories in the queue. Thus the story with the highest number/percentage of topical comments gets a bonus for being the most interesting story. Also a higher number (relatively) of editorial comments detracts from the score. Rather than using an absolute limit it seems that the rating of "interesting-ness" should be somehow determined by the community, and thus related to the rest of the stories that the community is reading.

[ Parent ]
Fix: Get the contribution of users, not comments (4.33 / 9) (#8)
by jesterzog on Sat Jan 27, 2001 at 05:36:34 AM EST

Considering it's usually diffucult to spread lots of spam or fake ratings with lots of different accounts, I wonder if there's a possible way around all of this by only counting the scores based on each user.

So if one user has posted 40 comments (probably spam), it'd only be the average of all the comments from that user - all of them put together would count as one unit.

This way, instead of measuring the value of comments in the story, we'd be measuring the vslue of each contributing user.

A similar workaround could be used for the ratings. Instead of counting each individual rating, average all the ratings from each user, add all these averages together and average it again.

There's at least one potential problem because if someone spams the story and lots of people rate a comment to 1 or 0, it's going to weigh down everyone's rating-making average. I'm not sure how to get around this but it might not be too difficult.


jesterzog Fight the light


[ Parent ]
move dropped stories to poster's diary? (4.72 / 59) (#11)
by danny on Sat Jan 27, 2001 at 07:15:01 AM EST

I suggested this is a thread somewhere else, but how about having dropped stories automatically moved to the poster's diary? That way comments on them won't be lost - and people who think a story is interesting but should have been a diary entry won't feel so bad about voting it down.

Danny.
[900 book reviews and other stuff]

Excellent idea. (2.33 / 9) (#12)
by Holloway on Sat Jan 27, 2001 at 08:56:00 AM EST

So do this, please.


== Human's wear pants, if they don't wear pants they stand out in a crowd. But if a monkey didn't wear pants it would be anonymous

[ Parent ]
Before you get too happy... (3.50 / 4) (#44)
by ramses0 on Sun Jan 28, 2001 at 11:26:46 AM EST

...Yeah, it's great entertainment for everybody else, but it's not what diaries were designed for. Read comment 41 for at least one dissenting viewpoint.

If you really like what a user posted as a diary entry, you can at the very least copy and repaste that story into the queue. But be careful, it might not be as popular an idea as you first think.

--Robert
[ rate all comments , for great justice | sell.com ]
[ Parent ]

and going the other way (3.88 / 9) (#29)
by danny on Sat Jan 27, 2001 at 08:15:46 PM EST

How about we have a "vote to move to section/front-page" option for every diary entry? (Maybe let people select which section they think it should go to.) If enough people think a diary entry is interesting (and/or there are enough highly rated topical comments), then it would probably interest people who won't see it unless it appears outside a diary.

Danny.
[900 book reviews and other stuff]
[ Parent ]

Make it a 'move to queue' option. (4.37 / 8) (#31)
by ZanThrax on Sat Jan 27, 2001 at 09:59:31 PM EST

Considering how few readers will read, and then vote on a given diary entry, the threshold will have to be very low. Automatically moving diaries to sections or the front page makes for incredibly easy abuse by multi-accounters and spammers that want to clutter up the main part of K5 with their crap.

Before flying off the handle over the suggestion that your a cocksucker, be sure that you do not, in fact, have a cock in your mouth.
[ Parent ]

yes, that'd work nicely (4.20 / 5) (#32)
by danny on Sat Jan 27, 2001 at 10:45:29 PM EST

You're right - moving exceptional diary entries to the submission queue would limit abuse. It'd also ensure that all stories go through the full scrutiny of the story queue.

Danny.
[900 book reviews and other stuff]
[ Parent ]

What section? And how do I veto? (4.71 / 7) (#41)
by ajf on Sun Jan 28, 2001 at 10:20:44 AM EST

I haven't used k5's diary facility yet, so I may be uninformed, but what section would you put a diary entry into?

And how do I prevent my diary entry from being submitted as a story by someone else, if I have some reason to do so?



"I have no idea if it is true or not, but given what you read on the Web, it seems to be a valid concern." -jjayson
[ Parent ]
well Interesting (3.00 / 8) (#13)
by /dev/trash on Sat Jan 27, 2001 at 11:22:08 AM EST

I always thought that teh system just 'timed' out articles after a certain time. Say after a week they were dropped.

---
Updated 02/20/2004
New Site
Unfortunately, it doesn't work (3.55 / 9) (#14)
by atom on Sat Jan 27, 2001 at 11:34:18 AM EST

Wow, first the first time, I received a message that said something like "You were the last person to vote on this article. A decision will be made, blah blah blah." Boy did I feel special. Then - wow, I voted on the next article, and it said the same thing! Today's my lucky day.

Unfotunately, both stories are still in the queue one over the threshold of 720 and one under. I think the system needs some tweaking - so that it works :) Unless there's an intentional delay in posting, then I think there's a bug. The articles I'm refering to are the last two in the queue - "Linux Co op Positions" and "Improving Linux."

Otherwise, it seems like a good idea, and will accomplish the desired effect.
dotcomma.org - Resource for programmers
Oops. (none / 0) (#22)
by rusty on Sat Jan 27, 2001 at 05:01:11 PM EST

See my update. I screwed up. :-)

____
Not the real rusty
[ Parent ]
Not entirely sure... (3.37 / 8) (#15)
by sugarman on Sat Jan 27, 2001 at 11:35:25 AM EST

I like the idea of setting a limit, btu I also have some reservations about posting / dumping stories based on a limit. Some good discussion could definetly get wiped out, and that often bugs me more.

Anyways, heuristics would be just part of the solution. I'm thinking a tweak to the basic post / dump threshold would aleviate a lot of the 'perceived' problem. The dupe and troll accounts, plus the inactive users have inflated the total above what seems able to control the queue.

The queue should really be as short as possible, to check for errors and validity, not begin and end the discussion. Be critical, is it good or not, and fire it through. If multiple queues are needed (one for editing, and one for ..."appropriateness"?), then that might help as well.

FWIW, most of the same has been said here, here, and here.

--sugarman--

heuristics and limits (none / 0) (#23)
by rusty on Sat Jan 27, 2001 at 05:06:53 PM EST

Well, the question is, would you rather we keep some of the stories that don't get posted via normal voting, or none of them? Basically, before this system, we kept none of them. Now we'll keep at least some of them.

The reason I want to go this way is because most of the time, on stories that are neither great nor terrible, the deciding factor really is the discussion it starts. I really want to be able to account for that in deciding what to do. This is almost definitely not the last change we'll make to this, but it's a start.

Another factor will be keeping track of "Active" users (instead of all users) to make the thresholds more accurate, and adding better tools for authors, like the ability to easily edit and resubmit, or just drop their own stories. I'm still planning on an editng area as well, so don't think I've abandoned that.

____
Not the real rusty
[ Parent ]

Inactive Users (4.00 / 1) (#28)
by crankie on Sat Jan 27, 2001 at 07:33:44 PM EST

I'd probably be one of the afore mentioned "Inactive Users" insofar as I read k5 when I'm waiting on a slow compile. So I read the articles, rate the comments and occasionally post a comment.

OTOH, I rarely have time to browse the story queue and so I appear to be a statisitical anomaly. Am I screwing the system and how can I avoid this?

~~~
"The great thing about hardcore socialists is the silence they emit once they start earning a decent wage." - tombuck
[ Parent ]
not at all (none / 0) (#30)
by rusty on Sat Jan 27, 2001 at 08:49:12 PM EST

I think an "inactive user" would be considered one who hadn't looked at the site for more than a month or so. Like, we'd have a "last visited" date attached to the user account, and whenever a user with a cookie loaded a page, that would get updated. Active users wold just be users whose "last visit" date was less than a month ago.

So, you have nothing to worry about. *Everyone* doesn't have to vote on stuff, just people who want to. And I think there are a lot fewer totally inactive accounts than people think there are. My guess is about 10% are probably really inactive, which doesn't affect the voting thresholds that much. OTOH, it would be nice if we could discount those from the threshold caclulation.

____
Not the real rusty
[ Parent ]

Time to inactive (none / 0) (#62)
by Mitheral on Wed May 29, 2002 at 03:38:50 PM EST

I think the inactive duration should be at least 45 days and preferably 90. For example, I get four weeks of vacation during which I'm unlikely to check K5 much if at all (I'm most likely to be found in the big blue room storm chasing, rallying, or offroading. All locations which are net impaired.) I'm sure I'm not the only one whose real life regularily disconnects me from the net.

[ Parent ]
After carefully going through the math... (3.00 / 14) (#16)
by Mantrid on Sat Jan 27, 2001 at 11:56:31 AM EST

After carefully reading Rusty's post and going through the math, all I have to say is:

"My cat's name is Mittens, his breath smells like cat food!"

ie, I don't really know what's going on, and as long as someone does I'm okay with it hehe. I tend to agree that something needed to be done...it wasn't too bad yet, but I don't think it was a self-correcting situation.




I guess it's too late to take my suggestion... (4.13 / 15) (#17)
by Speare on Sat Jan 27, 2001 at 12:35:12 PM EST

I should have suggested this earlier, perhaps, but...

I think all the queue needed was for implied voting. That is, if a story is generating lots of discussion, then one can infer that it rates higher. If it's generating lots of editorial postings, then conversely, something is likely wrong with the posting. If these inferences are collected after one day, it could help decide the submission in a more reasonable time frame.

So, as an example,

    Story begins with zero. Story needs ~100 to section, ~(-20) to die.

    First couple hours generates about six editorial top-level comments. "Resection this!" etc. This rates -6 implied, but we still have the rest of the day.

    Follow-ups to the editorials are not counted, as they are not easily inferred: they are often "I agree, fix this," and they are just as often "Nah, you're missing the point."

    After the would-be grammarians die down, the real discussion seems to kick in. Each top-level thread could be inferred as a +1 front page, and each follow-up to it as a vote to +1 section. This means that if the story delves deep on a couple issues, it's interesting to those few. If it keeps spurring new top-level comments, it has broader appeal.

    Safeguards must be considered, of course.

    Implied votes are kept separate from real votes, but are still tied to the person responsible for the vote, for reasons to be explained shortly.

    Implied votes are only tallied at the 24hr mark, and turned into real votes at that time (if the story is still in the queue). It is intended to spur a languishing vote by inferring the discussion value of a story.

    If someone gives a real vote, it nullifies any implied vote. This can be done since the implied and real votes are kept with user-ids.

    If someone posts more than once, it only counts as their most positive implied vote. This can be done since the implied and real votes are kept with user-ids.

    Aanyway. Let's hope Rusty's new scheme works at least as well as this scattered idea. :)


    [ e d @ h a l l e y . c c ]
Hmm (3.50 / 2) (#55)
by Elendale on Sat Feb 10, 2001 at 02:59:53 PM EST

Each top-level thread could be inferred as a +1 front page, and each follow-up to it as a vote to +1 section

One small point is that longer threads usually mean more 'disucssion' of ideas. This means that stories that create more 'discussion' would get sectioned more often. An interesting idea, but i like Rusty's better :)

-Elendale
---

When free speech is outlawed, only criminals will complain.


[ Parent ]
Story Status while its in the Q (4.05 / 17) (#19)
by citizenzero on Sat Jan 27, 2001 at 01:54:24 PM EST

Since we are introducing slightly more complex (and perhaps for a while, changing) limits in the queue, and since most people don't want to do all the funny math, I suggest that each story get some type of status indicator in the queue list.

Something like { new story | in voting | about to be dumped | about to be posted to front page | about to be posted to section page | about to be posted via Q limit | about to be dropped via Q limit }.

The rationale is that if people can see the stories current status they will be more likely to actually vote on it, especially if they like it and its nearing the brink of existence due to a timeout in the queue or something like that. Also, with all these changes to the system, its becoming more time consuming to figure out the status manually, and its good to give everyone as much information as possible (as we currently do by listing all the names of the people who voted for, against, etc.).

The list could also be made just to include the `negative' status { about to be dumped | about to be dropped via Q limit }, if people don't really care about the other ones.



I think (3.50 / 2) (#54)
by Elendale on Sat Feb 10, 2001 at 02:29:13 PM EST

It should only be visible after voting. Similar to how you show actual votes. Showing it before voting might skew the process (though it may not)

-Elendale
---

When free speech is outlawed, only criminals will complain.


[ Parent ]
Visibility before voting (none / 0) (#64)
by pediddle on Wed Jan 22, 2003 at 03:34:19 AM EST

But not showing it before voting would defeat the purpose of encouraging people to vote on almost-expired stories.  To avoid skewing the result, it could just show:
  • New story
  • Voting in progress
  • About to exit the queue
  • Old story (about to reach the voting limit)
That way there are no positive or negative preconceptions.

[ Parent ]
Cull the userbase? (3.81 / 11) (#21)
by Moneo on Sat Jan 27, 2001 at 04:00:13 PM EST

I think this problem is also partly the result of an increase in the number of percieved users (which increases the threshhold for story posting). There may be many accounts created by people who have gotten disinterested in k5 (or were created as throw-away accounts/trolls/etc.) Maybe you should introduce a TTL for users: if an account hasn't been used to post, vote or rate in N days/weeks/months it is disactivated -- that should make the total number of percieved users more closely approximate the number of actual participating users.
Propaganda plays the same role in a democracy as violence does in a dictatorship. -- Noam Chomsky
What if they never participate (4.00 / 6) (#24)
by MrSpey on Sat Jan 27, 2001 at 06:19:45 PM EST

I have an account on slashdot. I have it solely to customize the site (what the front page looks like, the display of comments, etc.). If someone has a similiar account on K5, one that exists just to get the same preferred layout all the time, then under your suggestion it would be deactivated after a fixed amount of time. This wouldn't be good. Even if we just delete users that haven't logged in for x number of time units then we get into the issue of people's accounts going away when they either can't use them due to real life constraints or they just don't bother logging in for a while. I think the best solution is to keep all accounts indefinitely but only count ones that are active at all when scoring votes for a submission. That way I can not use K5 for a few months and not lose anything, but at the same time I don't inflate the votes needed for a submission to get accepted or rejected.

Mr. Spey
Cover your butt. Bernard is watching.

[ Parent ]
You're right (3.75 / 4) (#26)
by Moneo on Sat Jan 27, 2001 at 06:50:58 PM EST

I don't customize the layout of k5 or /., so I hadn't thought of that. You're right though, and your suggestion accomplishes the same thing without recourse to deleting account.s
Propaganda plays the same role in a democracy as violence does in a dictatorship. -- Noam Chomsky
[ Parent ]
What about cookies? (3.75 / 4) (#34)
by Anonymous 6522 on Sun Jan 28, 2001 at 01:50:50 AM EST

Couldn't k5 just use cookies for display preferences? This would eliminate the need for people to make an account just for display settings, but I don't think there are that many accounts like this on k5. On Slashdot, the only thing you can do with in account is post comments, or moderate occasionally. On k5, you can always moderate and you can vote on stories in addition to posting comments. If someone has an account here, they will probably vote on stories and/or moderation occasionally. Heck, that big red (X new) should annoy them into voting. Basically I can't see anyone making an account here for the sole purpose of setting display preferences.

[ Parent ]
Re: What about cookies? (4.33 / 3) (#40)
by ajf on Sun Jan 28, 2001 at 10:20:36 AM EST

I assumed k5 already uses cookies for display preferences, because occasionally my comment display/sort options are "forgotten", but my login isn't. Might be faulty reasoning on my part though.

(As an aside, with slashdot, other uses of accounts are to configure the boxes down the right hand side, and to exclude sections or editors you don't like from the front page.)



"I have no idea if it is true or not, but given what you read on the Web, it seems to be a valid concern." -jjayson
[ Parent ]
What you want (3.00 / 1) (#56)
by fluffy grue on Wed Feb 20, 2002 at 12:36:09 PM EST

I believe that scoop uses a cookie for comment preferences if you haven't configured them directly, in which case it's definitely stored in your account.
--
"Is not a quine" is not a quine.
I have a master's degree in science!

[ Hug Your Trikuare ]
[ Parent ]

And quasi-lurkers... (3.00 / 2) (#36)
by Thaeus on Sun Jan 28, 2001 at 03:25:22 AM EST

...who occasionally get the urge to post a comment (this makes two for me in the half-year I've been reading K5). I'd prefer not to go to post a comment and discover that my account has lapsed.

thaeus, whose user info is now invalid. Ugh.
[silly sig search sounds super!]


----
*click*
----


[ Parent ]
Some problems with culling the userbase... (4.12 / 8) (#35)
by Anonymous 6522 on Sun Jan 28, 2001 at 01:59:27 AM EST

What happens when someone has their account deleted and someone creates a new one with the same name? Do the previous users comments get deleted, or will there be two sets of comments by two people with the same name? How would people tell between them? I don't think that using the UID would work because who wants to remember their UID or someone else's?

I just think that having if you having posted/voted/etc for a certian amount of time, your account is considered inactive and it won't be considered when the thresholds are computed.

[ Parent ]

Please do not do what is suggested (3.75 / 4) (#39)
by ejbst25 on Sun Jan 28, 2001 at 09:41:54 AM EST

I don't post very often...and I used to never post. I just like reading stuff...but why should I have to do anything but login and/or visit the page logged in in order to keep my membership? Mandatory participation does not seem like an intelligent way to do things.

-E

[ Parent ]
Rate (none / 0) (#61)
by Mitheral on Wed May 29, 2002 at 03:26:52 PM EST

Even if you are just reading you should still rate comments, at least occasionally. Ideally everyone would rate every comment they read.

[ Parent ]
In response to other responses: perception (4.50 / 4) (#43)
by Michael Leuchtenburg on Sun Jan 28, 2001 at 11:07:19 AM EST

I don't think Moneo was suggesting deleting actual accounts. But even if he was, here's a culling method that would work: don't delete the accounts, just don't count them for the purpose of voting threshholds etc. Of course, that requires completely changing the metric behind the voting threshholds, which are geared towards the number of people who vote in a given userbase (apparently the post threshold is around 0.9%).

I don't think this is needed, but it could be interesting. I doubt there are any people who vote who don't comment. Another interesting TTL possibility would be based on the last time the account voted on a story. In this way the voting mechanism would dynamically adjust the the amount of actual voters, rather than an estimate of the projected voters. Hmm, maybe average the past 20 stories votes/users_at_the_time and use that as the number of projected voters (for purposes of threshholds)? Or more. It would be calculated at story creation time, of course.

This would prevent display of threshold on the voting page, but what're you going to do? You could still have the threshholds displayed on a per-story basis.

[ #k5: dyfrgi ]
[ TINK5C ]
[ Parent ]

Some ideas (4.00 / 10) (#25)
by onyxruby on Sat Jan 27, 2001 at 06:41:42 PM EST

I've noticed some stories that were in the queue, generated a large number of votes (<500), and have scores that hover around 10 or so. If enough people care about a story one way or another to vote on it, why not post it by virtue of that fact alone? You could also do something similiar by the number of threads that come off of it. Trying to judge by the ratings that the individual comments get isn't really fair to the story poster. In short, if a story has generated "X" quantity of discussion and or votes, it has struck a nerve and has already proved itself relevant to k5.

The moon is covered with the results of astronomical odds.

"Apathetic Voters" section (3.62 / 8) (#33)
by Anonymous 6522 on Sat Jan 27, 2001 at 11:23:52 PM EST

Perhaps when the story reaches the the "maximum votes" threshhold, and it has a certain number of comments, it could be posted to an "apathetic voters" section. This would show that we couldn't make up our minds on the story. Stories that have more 0 votes than the sum of the +1 and -1 votes could also be posted to this section because most of us didn't care.

I second this (3.83 / 6) (#37)
by mumble on Sun Jan 28, 2001 at 04:54:31 AM EST

Introduce 2 more sections, one called "zombies" for stories that aren't quite alive, yet not quite dead either, and one called "graveyard" where all the dropped stories get sent, and everyone is afraid to visit. Though this might waste quite a bit of HD space.

Also, be careful how you judge "inactive" accounts, especially if you are going to start deleting them. Some people lurk and only rarely log in and use their accounts.

-----
stats for a better tomorrow
bitcoin: 1GsfkeggHSqbcVGS3GSJnwaCu6FYwF73fR
"They must know I'm here. The half and half jug is missing" - MDC.
"I've grown weary of googling the solutions to my many problems" - MDC.
[ Parent ]

Story Trolling (4.33 / 3) (#42)
by Michael Leuchtenburg on Sun Jan 28, 2001 at 11:01:10 AM EST

If dropped stories were kept (and displayed), trolls would flood the queue.. and the stories would remain. This can be solved by admin intervention, but the whole point of the K5 rating/voting system is to remove the need for admins as much as possible. It should run cleanly and unsupervised for years.

[ #k5: dyfrgi ]
[ TINK5C ]
[ Parent ]
These sections exist (4.33 / 3) (#51)
by Erf on Mon Jan 29, 2001 at 03:09:27 PM EST

The moderation queue is, by definition, full of stories that are neither alive nor dead. There is a "graveyard" as well, as far as I understand; judging by the K5 FAQ, stories are never deleted, and if you know the address you can view them (but of course they're never linked from any of the other stories pages). I'm not sure I see the value in letting everybody view the dumped stories -- I mean, they were dumped for a reason.

-Erf.
...doin' the things a particle can...
[ Parent ]

Not the front page... (3.66 / 6) (#38)
by B'Trey on Sun Jan 28, 2001 at 07:50:01 AM EST

Finally, we average those two scores together, to get a final score for the story. This will also be between 1 and 5. If it's above a certain value, the story will be posted to section-only. If it's above a higher value, the story will go on the front page.

I don't think any story which is not voted there should be on the front page. Set a single threshold for the story to make the section page.

bug in the system? (4.00 / 5) (#45)
by yavor on Sun Jan 28, 2001 at 12:29:32 PM EST

The new voting system looks interesting, but maybe there is a weak point. Votes on topical comments are counted. This doesn't look bad at first but consider this scenario:
Story A is posted for voting
Several malicious users really don't like the story
These users vote the story down but they
   (1) vote all topical comments down too.
   (2) write trollish topical comments so that other users vote them down.
This way the story would be dropped fast.
They can't vote a story up so easy because it is harder to write reasonable comments than writing something stupid. However they can still vote all topical comments up and increase the score.
There is an possible anti-attack - if users see that all topical comments are strangely molderated down(or up) they can vote them a little higher(or lower) than normal to make up for the malicious moderations. But again there is a possible anti-anti-attack - if the malicious users mod to 5(1) and then the other users vote them a little lower(higher) the bad guys can come back and change their moderations to 1(5) and make the effect even more powerful!

I don't think that this is going to happen soon because the majority of K5 readers are good users but this is a possible flaw in the system. But if this starts happening Rusty will be here to change the system again.

Scroll up/down (4.50 / 2) (#46)
by leviathan on Sun Jan 28, 2001 at 07:44:07 PM EST

While abusing corrective voting by changing your vote after the event is always a potential problem, see this comment for coverage of your other points.

--
I wish everyone was peaceful. Then I could take over the planet with a butter knife.
- Dogbert
[ Parent ]
Don't care. (4.11 / 9) (#47)
by Zer0 on Sun Jan 28, 2001 at 09:10:48 PM EST

Some users who really "Don't care" will now not vote at all. As "Don't care" should really be neutral and not negative. Maybe it should be changed from "Don't care" to "I'd rather it not be posted, but i'm not to fussed".

The way i see it now, not voting at all is a more effective "Don't care".

"Eh?" (3.00 / 4) (#49)
by Paul Dunne on Mon Jan 29, 2001 at 06:10:18 AM EST

> Nothing like alpha-testing on the live server, eh? :-)

So Canadianism is catching?! Everybody stay away from Inoshiro!
http://dunne.home.dhs.org/

what should have happened... (3.83 / 6) (#52)
by cetan on Mon Jan 29, 2001 at 03:45:13 PM EST

It's quite simple, the queue should have become a section. Either that or if an item is in the queue for too long, it goes to a new section: Limbo! :)

Joking aside, I think these changes will be good for K5. 2 days into the "experiment" as I type this and things seem to be much nicer in the queue.

===== cetan www.cetan.com =====
Limbo (4.00 / 1) (#60)
by emmons on Wed Mar 06, 2002 at 07:16:18 PM EST

Or, there could be a limbo section for every section that the story would have gone into. Just a link at the bottem or something- for all the articles that reached a threshold (votes or time).

This could also be done for 'dropped' stories. Not much good would be there, but if people really wanted to muck around in the muck, they could.

---
In the beginning the universe was created. This has made a lot of people angry and been widely regarded as a bad move.
-Douglas Adams

[ Parent ]
I don't think comments should have an effect (4.00 / 3) (#57)
by Hopfrog on Sun Feb 24, 2002 at 05:13:24 AM EST

Comment ratings having an effect on the story are a bad idea.
You are using a factor that can vary from being relevant to not at all being relevant to a story to rate a story.
Example: Somebody posts a story about the caste system in india. The first 2 posters makes some flamish comments about racism. The comments that follow degenerate to racist name calling, and these comments are voted down. The story suffers, though it might have been good.
Similarly, someone writes a story which is factually off track. The comments which point out the errors are all voted up. The story gains.
I think you should make it a bit simpler and give a story points when it generates a lot of topical discussion, regardless of the voting scores of the comments. This is a discussion site after all, and not an information site.

Hop.

An example why the new scheme is totally off (3.50 / 2) (#58)
by Hopfrog on Sun Feb 24, 2002 at 02:49:48 PM EST

This story reached the threshold, and was submitted.
A story of mine, now deceased, originally at this link, reached the threshold with scores that where above what the above story had.
However somebody posted some Osama Bin Laden comment that got him 20 1s.
I assume that is why my story got hidden and the other story not.
Such variables are unhealthy, and will foster ill will among your story submitting serfs.

Hop.

Vote for "maximum" and all other arbitra (3.40 / 5) (#59)
by johwsun on Tue Feb 26, 2002 at 03:16:47 AM EST

>So the first change is that in additon to the >normal post and hide thresholds, there's now >a "maximum votes" one as well. Right now, it's >set at 6% of users, or about 720 votes.

Because this 6% you put is arbitrary, I propose you to cast a permanent ballot for this and all other arbitrary values you set on kuro5hin.

Let us vote between 0% and 100% and the result will be the average(the mean) of all people votes.

saving comments (4.00 / 1) (#63)
by cronian on Sat Dec 07, 2002 at 07:13:19 PM EST

I think we need a way to deal with stories that generate good comments but are badly written. We should have a mechanism to make editorial changes to article after it has already entered the queue. We should have a new feature where users can actually rewrite and be voted on to replace the current article with a new one, but the comments would still be maintained. Thus, articles could get rewritten in a collaborative way before submission.

Along these same lines the edit queue could be modified to where users actually propose changes that are voted on. These changes could even be categorized so that grammar would be under its category.

Another thing that would improve the queue would be to have a wiki web edit queue where users could collaborate in creating articles before their actual submission and the user posting to this place woudl have control over submissions. It would provide a mechnaism for people to submit articles who can't write but still want to write about something.

Finally, I think it would be better to look at how many comments are highly rated rather than the overall quality of comments in general. If there are lots of great comments then the story is more worth saving than if there are only a few highly rated ones regardless of what the comments are overall. Something should also be done to not give users greater weight just because they rate more comments. Maybe you could only have a few of each raters' ratings count for each page.



We perfect it; Congress kills it; They make it; We Import it; It must be anti-Americanism
Story Queue Changes | 63 comments (63 topical, 0 editorial, 0 hidden)
Display: Sort:

kuro5hin.org

[XML]
All trademarks and copyrights on this page are owned by their respective companies. The Rest © 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!