Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

[P]
Maximum Votes Threshold Considered Harmful

By TheophileEscargot in Meta
Mon Feb 25, 2002 at 08:52:55 AM EST
Tags: Kuro5hin.org (all tags)
Kuro5hin.org

Everybody knows how a story gets posted on K5: its votes reach the post threshold, right? Wrong. About half of the stories posted never reach that threshold. It's even possible for a story to get posted with a final score of zero.

This is a Bad Thing. Here's why.

Update [2002-2-25 13:10:48 by rusty]: In response to this story and ensuing discussion, a couple changes have been made. See update at the end for a full description.


Originally on K5 a story had to reach the post threshold to get posted, but as volumes grew this led to stories hanging around interminably in the queue. It's reported that there were over forty stories in the queue, some of which had been there for weeks. In response Rusty made a change, introducing the Maximum Votes threshold. When the total number of votes reaches this (currently 350), then an algorithm is used to decide whether the story is posted. The number and the rating of the topical comments is taken into account, and the algorithm decides whether to post the story.

Reasons This Is Bad

  1. Button-pushing
    Trolling is a well-known way to attract lots of replies to a story. The famous world's most commented weblog story is a good example, currently having over 4,600 comments. Because the decision algorithm is based on the number of topical comments, amongst other things, the maximum votes rule has made it much easier to get a troll article posted on K5. By pushing the right buttons to attract lots of hostile comments, a poor story can more easily be posted this way.
  2. Time zones
    Different audiences for K5 tend to come on at different times of day. There tend to be peaks at about lunch-time for the different time zones of K5 readers. Formerly, because these marginal stories hung around in the queue, everybody had a chance to vote on them. Now that a story can get voted up in a few hours, with only a tiny number in favour, it's possible for a narrower group to get a story posted. For example this story had a majority of only 15, but was posted in about four hours.
  3. Cliques and Obsessive-Compulsive Reloaders
    Before the maximum votes threshold, marginal stories tended to hang around, so that a larger part of the K5 community was able to vote on them. Now marginal stories can be posted when only a much smaller section of the community has voted on them. I suspect the change has given a much stronger voice to that group slashdot calls "Obsessive-Compulsive Reloaders". We-- er, I mean they, constantly check K5 and weigh in with their votes immediately. Someone who only checks K5 once a day will probably not have the chance to vote on these marginal stories anymore.
  4. Statistics
    To get some idea of how many stories are posted this way I investigated the 20 stories on my "Everything" page. The raw data was posted in an earlier comment. The results were that 11 stories were posted by reaching the post threshold, and 9 stories were posted by reaching the maximum votes threshold. Their final scores were: 83, 71, 62, 64, 55, 25, 15, 13, 3. This shows that a significant number of stories are being posted this way, and thus that the maximum votes threshold has a significant impact on K5.
  5. Changes
    Since the Maximum Votes threshold was introduced, the number of users has increased dramatically. It is now much easier to get a large number of topical comments on a story. Anecdotal evidence suggests that has also been "rating inflation", where comments are rated higher; that it is now easier for trolls to provoke a response; and that there are more people posting editorial remarks in topical comments. In addition, the Maximum Votes threshold was originally set at 720, or 6% of registered users. Assuming 25,000 users, it is now at about 1.4%.
Proposed Solutions
Thanks to the change that keeps the voting stats permanently visible, it's now easy to see which stories got in through hitting the maximum votes threshold, not the post threshold. Have a look around, and judge for yourself. While only about half of the stories are posted this way, these stories are the ones where it is most important that the opinions of the whole community are taken into account, since it is debatable whether they should be posted. Several solutions have been proposed.
  1. Dump everything at the Maximum Votes threshold
    This is the solution that I now prefer. This would keep the queue from becoming crowded. It would encourage higher quality from authors, and would encourage more resubmissions. K5 has now grown to the extent that there is no problem with a lack of proposed stories, but there is a problem with poor quality stories. I think that this is the best way to maintain quality on K5.
  2. Abolish the Maximum Votes threshold
    The threshold could be abolished altogether, and stories should hang around as long as it takes. While this caused problems in the past, since then the readership has grown while the post threshold has been reduced, so it would not necessarily cause as much lingering as before. This solution is the most democratic, but would increase the size of the queue to some degree. DesiredUsername has pointed out that this would also help prevent repetitious stories being submitted many times in a short period.
  3. Put a floor in the algorithm
    This was proposed by makaera. The algorithm could be changed so that it only allows through stories with a final score above a certain level, possibly 20.
  4. Make stories wait 24 hours
    Localroger has also proposed that the story should wait 24 hours in the queue before the algorithm kicks in. That way every time zone gets a vote, and people who read K5 once a day get a chance to see the debatable stories. However, this option may be more difficult to implement in Scoop than the others. A similar "timeout" suggestion was made by Ranieri, suggesting that a story be give three days to reach the post threshold, then dropped.
  5. Change the thresholds
    I am Jack's username, amongst others, has suggested that the Maximum Votes threshold should be a percentage of users, not a fixed number. Alternatively, the existing thresholds could be modified to make it harder for a bad article to be posted.
  6. Rate stories as comments are rated
    A certain user says:
    Maximum votes is a patch to a troublesome implementation of voting in the first place. It shouldn't even be necessary at all, so I'm fairly resistant to trying to "perfect" it. What we should be doing is rating stories, like we rate comments, and deciding based on that. Rating is convergent, and tends to settle to a decision.
  7. Rate down topical comments in the queue
    This was jokingly proposed by hulver (who has a uid of 14, though he doesn't like to talk about it). If enough people were to systematically rate down topical comments in the queue, the stories would be less likely to be posted. This has the advantage that no code changes would be required.
  8. Leave the system as it is
    This option is fairly self-explanatory.
So the question is: what do YOU think? After all, YOU choose the stories!

Update [2002-2-25 13:10:48 by rusty]: Changes:

First, there's a new "voting floor," currently set to 5. If a story's score is below this, it won't be auto-posted. The actual value can be changed as needed.

Second, the auto-post trigger is now time based instead of votes-based. Currently it's set to 36 hours. That is, a story will hang around until it hits a post or drop threshold, or 36 hours have passed, whichever comes first. Essentially, this is "The localroger/Ranieri Solution", number 4 above. We shall see how it goes. :-)

Sponsors

Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure

Login

Poll
What should be done?
o Dump everything at the Maximum Votes threshold 19%
o Abolish the Maximum Votes threshold 9%
o Put a floor in the algorithm 10%
o Make stories wait 24 hours 34%
o Change the thresholds 9%
o Rate stories as comments are rated 7%
o Rate down topical comments in the queue 1%
o Leave the system as it is 9%

Votes: 88
Results | Other Polls

Related Links
o Slashdot
o Scoop
o zero
o reported
o Rusty made a change
o Trolling
o world's most commented weblog story
o this story
o raw data
o change
o DesiredUse rname
o pointed out
o proposed
o makaera
o Localroger
o also proposed
o "timeout"
o Ranieri
o I am Jack's username
o suggested
o A certain user
o says
o jokingly proposed
o hulver
o Also by TheophileEscargot


Display: Sort:
Maximum Votes Threshold Considered Harmful | 90 comments (85 topical, 5 editorial, 0 hidden)
Implement solutions 3, 4 and 5 (4.66 / 3) (#3)
by grand master thump on Mon Feb 25, 2002 at 06:55:02 AM EST

Implementing solutions 3, 4 and 5 seems the most sensible move, allowing the combination to work together. Putting a floor on the votes for the algorithm is sensible to stop the trolls (if thats what is wanted). 24 hour delay giving all of the regulars a chance to put in their tuppence worth. And by having the max votes as a percentage of the user base allows the algorithm to scale with the users (though in time to come this will be discussed again as accounts die out through natural wastage and the percentage does not reflect the true active user base).

Though I think just a floor on the algorithm is the most important, if a story is provoking discussion the discussions are often worth reading even if the story is not, but real dross and slanging matches need to be weeded out.


This sig has been stolen

Natural wastage (3.00 / 2) (#9)
by jonathan_ingram on Mon Feb 25, 2002 at 08:07:47 AM EST

And by having the max votes as a percentage of the user base allows the algorithm to scale with the users (though in time to come this will be discussed again as accounts die out through natural wastage and the percentage does not reflect the true active user base).

This is easily dealt with by making the max votes a percentage of the 'currently active userbase', which could be determined once a month/week/day/minute :) -- probably by counting those user ids which had been active over the last month.

Given the large amount of transparency in the kuro5hin story voting process, it would also be nice if voters were informed about the maximum vote threshold before they voted, just as they currently are about the post/hide thresholds.
-- Jon
[ Parent ]

Fix Fix Fix (3.00 / 3) (#10)
by Ranieri on Mon Feb 25, 2002 at 08:09:44 AM EST

I consider 3 and 5 to be fixes to a fundamentally flawed system. This main flaw of the system is opacity, i.e. lack of transparance. I think it's better for all people involved if the rules followed by the algorithm are not only effective (which apparently they are not) but also simple and verifiable.

Transparent situation: A story gets posted when it reaches the post threshold.

Non-transparent situation: A story gets posted when is reaches the post threshold, or when it reaches the maximum votes threshold and the current score is above a certain floor and the sum of the comment moderations is above a non-disclosed threshold and a wide range of other unspecified conditions are met

You can tweak the parameters all you like, but in my opinion it's a complicated and obscure way to do it, and it brings the site no benefits whatsoever. It was introduced to keep stories from ligering for weeks in the queue. There are other ways of tackling this problem that are simple, transparent and actually manage to keep some of the crud off the front page.
The Maximum Votes system must die.
--
Taste cold steel, feeble cannon restraint rope!
[ Parent ]

Such as? (3.33 / 3) (#24)
by rusty on Mon Feb 25, 2002 at 10:27:45 AM EST

There are other ways of tackling this problem that are simple, transparent and actually manage to keep some of the crud off the front page.

And your solution is?

____
Not the real rusty
[ Parent ]

My propositions was .. (4.00 / 3) (#30)
by Ranieri on Mon Feb 25, 2002 at 11:00:03 AM EST

A similar "timeout" suggestion was made by Ranieri, suggesting that a story be give three days to reach the post threshold, then dropped.

With rationale explained in this comment.

Regarding your suggestion to rate sories the way we rate articles, i think it might turn out to be an good discriminator. Often i find myself in a mood that is not "binary" enough to choose between "0" and "+1", usually ending in me voting one of the two pseudorandomly, or not voting at all. Being able to add more nuance to the vote will undoubtedly benefit the moderation process.
Controversial articles could potentially degenerate in "1" vs. "5" shouting matches, but as long as the new system is designed to reduce back to the current system for a binary voting style, it's guaranteed not to be any worse.
--
Taste cold steel, feeble cannon restraint rope!
[ Parent ]

Ranieri's solution... (4.00 / 2) (#31)
by TheophileEscargot on Mon Feb 25, 2002 at 11:02:10 AM EST

...was a three-day timeout, as described, credited and linked in "Solution 4".

Hmmm. Now if you want another anti-volume argument, how about "Even rusty doesn't have time to read an article before he comments..."

Just kidding! I know may seem like blatant ingratitude, but I'm not the only one to think that the system needs a tweak or two every so often. Is that so wrong? ;-)
----
Support the nascent Mad Open Science movement... when we talk about "hundreds of eyeballs," we really mean it. Lagged2Death
[ Parent ]

Ha! (4.66 / 3) (#35)
by rusty on Mon Feb 25, 2002 at 11:14:48 AM EST

Mea culpa. I did read the whole thing again before poting my long response. Not enough coffee yet when I posted that comment.

I'm not the only one to think that the system needs a tweak or two every so often. Is that so wrong?

Not at all. Sorry if I seem grumpy. I feel like I have a duty to put up some resistance to changes and see how serious you all are about them. :-)

____
Not the real rusty
[ Parent ]

An example on why Rusty's new scheme is off (3.00 / 4) (#5)
by Hopfrog on Mon Feb 25, 2002 at 07:22:38 AM EST

Right here

Hop.

"New scheme" (3.40 / 5) (#19)
by rusty on Mon Feb 25, 2002 at 09:47:09 AM EST

I love how it's new now that everyone knows about it.

The auto-post algorithm has been in operation since January, 2001.

____
Not the real rusty
[ Parent ]

Y'know (3.50 / 4) (#22)
by wiredog on Mon Feb 25, 2002 at 10:03:16 AM EST

Having the volume of complaints hit "critical mass" only once a year isn't that bad. And tweaking the system once a year isn't that bad either.

A'course, I'm not the guy who's gonna be doing the tweaking. I just sit on the sidelines and bitch about it.

Peoples Front To Reunite Gondwanaland: "Stop the Laurasian Separatist Movement!"
[ Parent ]

Heh. You and everyone else. (NT) :-) (3.66 / 3) (#25)
by rusty on Mon Feb 25, 2002 at 10:29:53 AM EST



____
Not the real rusty
[ Parent ]
Ghost polls (3.00 / 2) (#29)
by jonathan_ingram on Mon Feb 25, 2002 at 10:59:35 AM EST

<semi-OT>
Amusingly, I notice that while the story of yours that you link to was voted down, and so deleted, the poll that accompanied the story is still there - as are all the related links. So, I can see that it is a story about robot weaponry of some sort :).

Is there are way to get a list of all the current/recent polls, other than through the search engine? I don't seem to be able to find one - although polls are generated so often that it probably wouldn't be much use.

I also see that, given a poll, there is no simple link to click to get back to the story/diary entry which is associated with it.
</semi-OT>

You highlight an interesting point - which is that you can stop stories from being posted via the maximum post mechanism by trolling the story, causing a large number of low comment votes. It would be interesting to know if this is an intended side-effect of trying to take into account stories with good discussions.
-- Jon
[ Parent ]

Yeah (4.50 / 2) (#34)
by fluffy grue on Mon Feb 25, 2002 at 11:11:53 AM EST

See the "Other Polls" link below the frontpage poll.

Also, with a poll there is a way to get back; the story will be linked (with title) towards the left and down a little, right over the comments area.
--
"Is not a quine" is not a quine.
I have a master's degree in science!

[ Hug Your Trikuare ]
[ Parent ]

I want to root for my own proposition ... (3.60 / 5) (#6)
by Ranieri on Mon Feb 25, 2002 at 07:23:13 AM EST

... and to point out a suggestion made by wiredog in a comment:

What I suggested a year or so ago, after one of my stories languished in the queue for several days, was having a "time to live" variable set as part of the submission. Stories that would become outdated quickly could have a ttl of half a day, up to a max of a week or so. Maybe have the max ttl modified by the number of stories in the queue.

Note that this TTL feature can be seen as either a replacement or a complement of the recently proposed feature that allows authors to remove their story from the moderation queue.

I am of the opinion that the "maximum votes" feature is clearly not having the desired effect. Let's get rid of it and replace it with something more transparent.
--
Taste cold steel, feeble cannon restraint rope!

From looking at the stats... (3.33 / 3) (#11)
by wiredog on Mon Feb 25, 2002 at 08:20:54 AM EST

  • For: 82, 45 FP
  • Against: 8
  • Didn't care: 7

Don't recall the last time a story (except the Chuck Jones one) got that sort of proportion of post vs dump vs don't care. CLearly, fixing the queue is an Idea Whose Time Has Come (again...)

An earlier comment quotes part of my solution for the problem. A time to live, set by the submitter and modified by number of stories in the queue, plus an elimination of the voting threshold.

Peoples Front To Reunite Gondwanaland: "Stop the Laurasian Separatist Movement!"

Well (5.00 / 1) (#50)
by Anonymous 242 on Mon Feb 25, 2002 at 12:49:45 PM EST

There was that annoying Nobody expects the Spanish Inquisition! piece a while back:

Score: 80
Total votes: 86
Voted for: 82, 73 as FP
Voted against: 2
Didn't care: 2

There is no accounting for some people's tastes!

Regards,

Lee Irenæus Malatesta

[ Parent ]

Extremes in voting (4.00 / 1) (#68)
by I am Jack's username on Mon Feb 25, 2002 at 05:36:27 PM EST

From memory, articles with extreme voting patterns:

Lowest % of -1s:
All hail the Leonids meteor storm
Total: 88. For: 80 (63FP) =90.9%. Against: 0 =0%! Didn't care: 8 =9.09%.

Highest % of +1s, and lowest % active apathy:
Nobody expects the Spanish Inquisition!
Total: 86. For: 82 (73FP) =95.35%. Against: 2 =2.33%. Didn't care: 2 =2.33%.

OSDN to end k5 affiliate agreement
Total: 90. For: 83 (71FP) =92.22%. Against: 3 =3.33%. Didn't care: 4 =4.44%.

Maximum votes threshold considered harmful
Total: 122. For: 104 (54FP) =85.25%. Against: 9 =7.38%. Didn't care: 9 =7.38%.

Anyone know of a really controversial story where everyone either loved or hated it with 0 "Don't care" votes? Anyone with some calculated stats?
--
Inoshiro for president!
"War does not determine who is right - only who is left." - Bertrand Russell
[ Parent ]

My other "extreme" story... (3.00 / 1) (#69)
by TheophileEscargot on Mon Feb 25, 2002 at 05:51:58 PM EST

...was Deep Space 1 pictures of comet Borrelly.

It got in with 83 for, 3 against; but got no comments at all for a month and a half, when spiralx got finally got in a "first post".

I choose to interpret that as popularity. My story clearly said it all!
----
Support the nascent Mad Open Science movement... when we talk about "hundreds of eyeballs," we really mean it. Lagged2Death
[ Parent ]

Adjust threshold, test new voting systems (4.00 / 5) (#12)
by Eloquence on Mon Feb 25, 2002 at 08:39:48 AM EST

The problems you mention really do not relate to the threshold per se, but rather the way it is currently implemented. Let's go through your different points:

1) Button-pushing. You are correct, by weighing in topical comment scores, much-discussed troll stories have a certain likelihood to make it. Whether this is a good or a bad thing depends on what kind of site you want. I have always seen K5 as primarily a discussion site, where people go when they want to see all arguments about a given issue. By dumping stories that have lots of high-quality topical comments just because they are too controversial and don't make it within a short amount of time, we also stop the discussion (and make it mostly invisible). Note that the Adequacy story you mention would not have made it in that form, because of the large number of low-rated topical comments.

2) Time zones. Yes, the threshold could be higher and relative. A minimum time is not a good idea IMHO because it would hold back critical news, and K5 is slow enough as it is. By the way, given K5's nature, I would assume that many people's sleeping rhythms are far from regular, so people from all over the world are probably voting all the time.

3) Cliques and obsessive reloaders. Again, a higher threshold would mostly fix this problem, as far as it exists.

Your anecdotal evidence cited in 5) is a bit thin. I can see no such developments, and even if they exist, they do not necessarily relate to the threshold, for the exact reason you mention: The number of users is continually increasing and coming in from lots of different sources. How they behave is hard to predict. Your model can only be tested on empirical data, and has to prove its predictability on several sets of data. Otherwise it's just an opinion.

Solutions:

1) Dump everything. Not a good idea. The same thing you criticize -- button-pushing -- can backfire against you. If someone criticizes the "moral majority" on the site, his story may be quickly shot down by said majority. Much as I loathe deliberate trolling, the "dump everything" rule encourages groupthink.

2) Abolish threshold. You haven't been around so long. I've been here since early 2000 and I watched the inflation of stories in the queue. At some point, it got quite terrible. Stories were lingering for days or weeks and frequently you had a two-digit number of submissions pending. It was very unsatisfactory for authors and voters alike. People started to complain that they couldn't really be expected to vote on so many stories. ShouldExist is another Scoop site that worked without the threshold for a long time. There were about 30 stories in the queue last time I checked before they updated, many of them more than a year old. Yep, a year. On infoAnarchy I frequently push stories to the frontpage myself if they take too long.

3) Floor in the algorithm. This is really fine-tuning and the kind of stuff that's hard to explain a year after you've done it. I would not necessarily oppose this, but I would want to see actual evidence for harmfulness of the current solution first.

4) 24 hour story wait. No, for reasons explained above.

5) Change thresholds. Yes, this may well be worth doing. A relative threshold seems like a smart idea. This is what is generally used for real world direct democracy.

6) Rating stories. Won't really fix the problem on its own, you will still use some algorithm to decide whether a story has "stabilized" enough. But yes, this is a good idea that still needs to be tested in practice. It would be good if Rusty specified the whole system and submitted it to K5 before implementing it to get as much feedback as possible on potential problems.

7) Rating comments down. Rating should always be based on a comment's content, not on some desired story vote outcome. This is the kind of hacking that ultimately breaks moderation systems.

8) Leave system as it is. Well, not really -- we all want progress. But adjusting the threshold and then slowly moving to a new voting system seems like the smartest idea to me.
--
Copyright law is bad: infoAnarchy Pleasure is good: Origins of Violence
spread the word!

Note (2.66 / 3) (#13)
by Eloquence on Mon Feb 25, 2002 at 08:44:14 AM EST

In Solution 5), I'm referring to the maximum vote threshold. Modifying the actual post/dump thresholds based on user population has been done in the past and did not work so well; it would in fact lead to more stories being posted because of the maximum vote threshold.
--
Copyright law is bad: infoAnarchy Pleasure is good: Origins of Violence
spread the word!
[ Parent ]
My Comments (3.50 / 6) (#14)
by GreenHell on Mon Feb 25, 2002 at 08:55:05 AM EST

First, it definately needs some sort of change, but I'm not sure what the best method would be.

IMHO though, I don't think that all stories should just be dumped if they hit the maximum votes threshhold. Why? Well, the current implementation was done for a reason. Besides keeping the queue clear (which dumping them does as well), the Maximum Votes was also designed to preserve stories that weren't on the fast track to being posted, but had a lot of discussion going on. By dumping them, you lose a lot of the previous discussion, even if it gets reposted, it's doubtful everyone will want to refight all the same battles. So dumping's not really a good idea.

Abolishing the threshold. Hmmmm, interesting idea. Sure the readership may have grown, but there's still going to be stories that just seem to languish in the queue forever. Even if you got rid of the maximum vote, there'd still need to be some way to get stories out of the queue (by posting or dumping) that have been there just way too long.

Next up, we have a floor on the algorithm. A good idea, it keeps the stories that aren't doing to well from being posted. Problem is that setting it too high can have the same effect as dumping stories, and you still may lose large & interesting discussions.

Time. Well, this is personally the one I like the best, make a story wait a while before it gets posted. Only problem is that some stories shouldn't wait a while. Did we really want the Chuck Jones story to wait 24 hours before being posted? (Actually, that one didn't get posted via the maximum votes algorithm, so it doesn't really matter) But you get the idea, some stories just shouldn't have to wait. Now the other time suggestion is the exact same as the dump them all, only it sets the limit to time, not votes but still has all the same problems.

Percentage: Personally I seem to remember it originally being made as a percentage of the users. But it's probably a number that was calculated and then hardcoded in. This is probably the only suggestion without a downside on there. Changint he thresholds on the other hand is iffy. You don't want to make it impossible for something to get posted, but at the same time, it can't be too easy either.

Rate stories as comments are rated. Ugh... I'd skip voting on some stories if this was the case. What do you give instead of a '0 (Don't care)'?

Rate down topical comments in the queue. Jokingly? good, because all I can say to this is "Next!"

Leave the system as it is. Well, I really have no problem with the system, but people seem to think there is, so this probably isn't the best idea.

I dunno what's going to happen to it, or what should happen to it, but every suggestion is going to have problems with it. Maybe it should just stay the way it is.

-GreenHell
This .sig was my last best hope to seem eloquent. It failed.
User percentages (4.00 / 4) (#21)
by rusty on Mon Feb 25, 2002 at 09:53:54 AM EST

Percentage: Personally I seem to remember it originally being made as a percentage of the users. But it's probably a number that was calculated and then hardcoded in.

Actually, no, it's something Scoop does on the fly. Basically, when it checks the thresholds, it checks how many users there are, and then sets the thresholds according to what the admin sets the percentages to be. What we found a while back was that after a certain number of people, the percentage scheme basically just gets more cumbersome than it's worth. It was down to something like 0.01% and it was still too high, so I just set the thresholds to static numbers.

Leave the system as it is. Well, I really have no problem with the system, but people seem to think there is, so this probably isn't the best idea.

I'll take that as another vote for "It works well enough now." :-)

Seriously. A few people seem to have a problem with it. That doesn't necessarily make them right. My feeling is that there's no answer that will make everyone happy all the time, and the answer we have now is as good or better than anything else I've heard. So if you think there isn't a problem, please speak up.

____
Not the real rusty
[ Parent ]

User percentages (3.66 / 3) (#23)
by jonathan_ingram on Mon Feb 25, 2002 at 10:21:05 AM EST

What we found a while back was that after a certain number of people, the percentage scheme basically just gets more cumbersome than it's worth. It was down to something like 0.01% and it was still too high, so I just set the thresholds to static numbers.

This is 0.01% of all registered accounts? If you have 100,000 registered accounts, then that's 10 people :).

Of course, the real problem is that you don't want to count registered accounts - due to natural churn you'll have a lot of accounts that are no longer used, or domant (as mine was for a year or so). Do you have any way to generate statistics on the number of active accounts in any one month? These could then be used as the basis of your percentages -- and would have the useful effect of automatically adapting the thresholds to rises or falls in the activity level of your site.

Of course, there are different levels of 'active' - those that just read stories, but never vote or comment or submit, may not be counted as active for the purposes of this statistic.
-- Jon
[ Parent ]

Er (3.50 / 2) (#38)
by rusty on Mon Feb 25, 2002 at 11:22:56 AM EST

Um, I meant one tenth of all users. Not one tenth of one percent. Sorry. :-)

Of course, the real problem is that you don't want to count registered accounts - due to natural churn you'll have a lot of accounts that are no longer used, or domant (as mine was for a year or so).

Yes.

Do you have any way to generate statistics on the number of active accounts in any one month?

Sort of. Maybe. In a way. :-)

What we could do is have a daily (or weekly) cron which looks at sessions that have been active in the last $time_period, and counts the number of unique accounts represented by those sessions. This would be dumped into a Scoop var called "active_users" or something, and when counting percentages, that could be used as the current user count.

Might be worth trying.

____
Not the real rusty
[ Parent ]

Duh. Dyscalculus (3.50 / 2) (#39)
by rusty on Mon Feb 25, 2002 at 11:25:51 AM EST

Um, I meant one tenth of all users. Not one tenth of one percent. Sorry. :-)

That doesn't make any sense either. I don't know what the hell I meant. Please disregard all actual numbers, and just take away a sense that the percentage thresholds were not keeping good track of actual voting users. Trust me on this. :-)

____
Not the real rusty
[ Parent ]

Active users, "News" (3.00 / 2) (#66)
by I am Jack's username on Mon Feb 25, 2002 at 04:52:58 PM EST

Active users for this might be k5ers who voted for stories during the last 7 days. What about reducing the time limit for "News" stories?

These were mentioned in the comment Theo linked in sol5.
--
Inoshiro for president!
"War does not determine who is right - only who is left." - Bertrand Russell
[ Parent ]

Why is reducing amount of content is so important (3.77 / 9) (#15)
by brunes69 on Mon Feb 25, 2002 at 09:18:28 AM EST

....seriously? I constantly see these posts on K5, seeking to seperate the "wheat from the chaff" and "too much crap is being posted" etc. Here is a serious question for all those who feel this way: so what?

The solutions most seem to prefer is a higher degree of filter on the stuff that gets posted here. This won't spontainiously create more high-quality content, all it will do is decrease the overall amount of stuff on K5. What I propose - deal with it. If you start reading a story, and decide it is crap, stop. Just because it is of no use to you does not mean that it is of no use to someone else.

I mean, it's not like K5 is working with a finite amount of space on the front page for stories here. What is the big deal when a few get posted that don't meet your "high standards"? I for one would much rather have a site with vast amounts of content t, most of it good and some of it bad, than a site with a select few, hand picked articles. Once you start going down this road you end up looking alot like someone else.



---There is no Spoon---
Several reasons (4.66 / 6) (#17)
by Gerald Mouse on Mon Feb 25, 2002 at 09:37:53 AM EST

  • Too great a volume makes it hard to keep up. If you only check K5 every few days, you're confronted by a huge number of crap articles, and no easy way of telling which are the good ones without the tedious process of going in to each one. Doesn't matter so much to compulsive reloaders of course.
  • Greshams Law of Weblogs: bad content drives out good. When it becomes obvious that you can get a story posted without doing any real research, why should I bother doing research? Why bother considering other points of view and writing a careful balanced article, when a one-sided flame will get in just as easily? Why bother looking up multiple sources and opinions when you can just MLP to the Washington post?
  • If most of the articles become flames, then more flamers will join the site, while the non-flamers will go elsewhere. Some of those who left in disgust after September 11th have come back, but some haven't. Do we want a site by flamers for flamers?
Also, I don't see much evidence that the somewhere else you mentioned has too few, too high-quality stories. The problem with /. stories that I see are that they're short, badly researched, and unchecked. Trying to improve the story quality is going in the opposite direction to them.

[ Parent ]
by flamers for flamers (1.66 / 3) (#20)
by wiredog on Mon Feb 25, 2002 at 09:49:33 AM EST

That would be "naked.swishymen.org"

Peoples Front To Reunite Gondwanaland: "Stop the Laurasian Separatist Movement!"
[ Parent ]
Yay! (4.14 / 7) (#18)
by rusty on Mon Feb 25, 2002 at 09:44:07 AM EST

I constantly see these posts on K5, seeking to seperate the "wheat from the chaff" and "too much crap is being posted" etc. Here is a serious question for all those who feel this way: so what?

That's exactly how I feel. There are some submissions that are just a waste of everyone's time. Those get discarded. There are many that are appealing to a few people, and uninteresting to a lot of people. Those tend to be posted more often than not (via the auto-poster, usually). Then there are some that are clearly great stories, and those get posted quickly.

I like the way it works. So I'm not too likely to put a high priority on reworking it. Nevertheless, if someone wants to implement one or more of these solutions, and provide a convincing argument why they might be better, I'd certainly listen.

But like you, I don't know why reducing the number of stories is such a compelling goal.

____
Not the real rusty
[ Parent ]

yay. (4.00 / 2) (#26)
by imrdkl on Mon Feb 25, 2002 at 10:37:40 AM EST

While auto-post is a good thing, especially for those of us that tend to post boring articles, it could use at least a bit of tweaking.

Why not adjust the parameters in auto-post to make it equivalent to the recent increase in the standard post threshold? Since minimum post threshold went from 70 to 95, the corresponding increase should come in the auto-post.

I have complained in my diary about topical comments before a post makes it out of the queue, and the practice seems to be growing in popularity. Perhaps it's a mojo-grabbing thing, or perhaps folks are just very happy to discuss things, but these days, alot of the topical comments and moderation of them come in the queue.

My suggestion in Theo's diary was to bump the autopost limit to 450, and reduce the weight of moderation on topical comments for autopost.

It will be sad to see a good discussion disappear with a boring posting, but thats the risk you take when you make comments in the queue.

[ Parent ]

"What about Blaine?" (3.00 / 3) (#28)
by Pihkal on Mon Feb 25, 2002 at 10:53:50 AM EST

One thing I haven't seen mentioned so far is the possibility of altering how the system factors in comments when a post reaches the maximum vote threshold. How about a setup where weight is given to comments/threads that favor longer comments over shorter ones?

This has a couple of advantages. It would reduce the effect of one-line flames and little quips. Also, this would be a more accurate measure of good, lengthy discussions. One way might be a ratio of comment number to total comment length. Or better still, consider an exponential system, which would place much higher weight on the scores of long comments (e.g., multiply the score of a comment of over 200 characters by a factor of 4).

The biggest disadvantage I see is the difficulty of clearly seeing it in action, and the necessity of fiddling with the constants to achieve a good result. Still, I like how it would favor longer comments and downplay/ignore flames.

Or as another thought, how about requiring a minimum story length for everything but MLP?



"I will not be pushed, filed, stamped, indexed, briefed, debriefed or numbered!"
-- Number 6
[ Parent ]
Beware of exploitable systems (3.50 / 2) (#36)
by jonathan_ingram on Mon Feb 25, 2002 at 11:15:05 AM EST

How about a setup where weight is given to comments/threads that favor longer comments over shorter ones?

There isn't really a strong correlation between length and interest. Often the most useful comments in a thread are the short, informative ones.

If you give a bonus to long comments, then prepare for a repeat of the Slashdot crap-flooding wars. The next idea you'll have is to try and compress the text, and discard it if it's too compressible... that way lies madness. A complex system is far too much work to scale -- far better to have something simple, which is much harder to exploit effectively.

This is also the reason why I dislike the way that a rogue voter can negatively influence the posting of a putative story, by voting every topical comment down.
-- Jon
[ Parent ]

Not a good idea (4.00 / 2) (#37)
by rusty on Mon Feb 25, 2002 at 11:18:15 AM EST

All that would do would be to encourage long meandering comments (or stories) rather than good ones. "Length" alone is never a good indicator of quality, IMO. If you want to prioritize quality, we should be considering ratings in the way that we already are. That is, by looking at ratings, we're at least drawing our data from people's judgements, rather than an unsupportable assumption ("length == quality").

____
Not the real rusty
[ Parent ]
Aye. (4.00 / 3) (#16)
by THoliC on Mon Feb 25, 2002 at 09:37:36 AM EST

I wanted to to +1FP this, but unfortunately was offline when it was posted to the queue and subsequently (and correctly IMO) voted up.

As 'Wiredog' has mentioned, I don't think I've ever seen a story burn through the queue quite so fast. I was gratified to see that it made the front page on 'vote merit', thus avoiding the awkward and paradoxical situation of it getting posted on the 350 vote threshold rule which, whilst amusingly ironic, would have blunted the point.

I've said all I have to say on this in the diaries mentioned in the story itself, so I think I'll leave it there with an unnecessarily overblown and somewhat out of context quotation:
"The only thing necessary for the triumph of evil is for good men to do nothing."
Edmund Burke (18th Century)
So, Rusty, please do something...

T.


"Wanderlust,
has got us both,
looking for a bed today..."

I've re-considered your request (2.04 / 24) (#27)
by rutsy on Mon Feb 25, 2002 at 10:46:29 AM EST

Basically, I intend to make the following changes:

  • Reduce the post threshhold to 80 again, this will allow more stories to make the cut
  • Increase the floor to 0, so that bad stories don't linger in the queue
  • Randomly edit comments, adding secret messages directed at the Maine chapters of the Knights Templar
  • Rework the posting algorithm so the reciprical of the number of "editorial" posts divided by the number of "topical" posts posted before noon with an average comment weighting of at least 3.2522505 will be factored into a recursive differential equation where the sum of the squares of all the comment ratings by trusted users multiplied by the root mean squared of the weighted moving average-computed median will be discarded and I'll just go back to putting stuff in section/FP manually.

-----
Wouldn't it be great if everybody who owned a weblog was as great as me? -rutsy
Welcome! (2.71 / 7) (#32)
by jonathan_ingram on Mon Feb 25, 2002 at 11:05:54 AM EST

Note the name - 'rutsy' vs 'rusty'.

Welcome to uid 25743, the newest member of the Kuro5hin community... and trolling with his first comment :)
-- Jon
[ Parent ]

Are you sure you want to increase the floor to 0? (2.00 / 2) (#45)
by MmmmJoel on Mon Feb 25, 2002 at 12:21:39 PM EST

Increase the floor to 0, so that bad stories don't linger in the queue
Won't this mean that if a new story's first vote was a negative that it would be dropped from the queue? I thought that was the point of a negative floor to begin with. Perhaps just raising it would be a better option.

[ Parent ]
Heh (3.25 / 4) (#53)
by rusty on Mon Feb 25, 2002 at 01:05:41 PM EST

Look at the username carefully. Someone's having a bit of fun with you.

____
Not the real rusty
[ Parent ]
Yeah (3.66 / 3) (#54)
by wiredog on Mon Feb 25, 2002 at 01:11:05 PM EST

Because we all know that Rusty's trolling account is really this one.

Peoples Front To Reunite Gondwanaland: "Stop the Laurasian Separatist Movement!"
[ Parent ]
The Long Response (5.00 / 8) (#33)
by rusty on Mon Feb 25, 2002 at 11:11:41 AM EST

Various and sundry responses

Now that a story can get voted up in a few hours, with only a tiny number in favour, it's possible for a narrower group to get a story posted. For example this story had a majority of only 15, but was posted in about four hours.

Was it really only 4 hours to hit 350 votes? If so, then the maximum votes should be raised. It was originally targeted for about 36-48 hours.

This shows that a significant number of stories are being posted this way, and thus that the maximum votes threshold has a significant impact on K5.

Did you work out any estimates like "If the floor was X, or the minimum ratuings were Y, 9 of those 11 wouldn't have been posted..."? I know a lot of stories get auto-posted. I don't know if there's a clear division between the ones that do.

In addition, the Maximum Votes threshold was originally set at 720, or 6% of registered users. Assuming 25,000 users, it is now at about 1.4%.

I don't remember it being 720. That could be true, though. There's a lot I don't remember. :-)

Proposed solutions

To look at your proposed solutions individually (and thank you for suggesting some solutions, at least, unlike the majority of meta-articles which just provide complaints. ;-)):

  1. Dump everything at the Maximum Votes threshold

    I think this would be a bad idea, for a couple reasons. First, one hallmark of stories that are good, but don't get voted up overwhelmingly is controversy. I think that if we went with a firm "votes-only" policy, we'd end up with no articles that really challenge us with a new perspective. Saving stories that are controversial, while keeping queue times down are the main goals of the auto-post system. I think dropping everything below the thresholds would lead to a much less interesting site.

  2. Abolish the Maximum Votes threshold

    Not gonna happen. Well, to generalize, we will not return to an open-ended system. If many of you either weren't here or don't remember it, it sucked. The complaints were far more vocal and constant then they are about this.

  3. Put a floor in the algorithm

    There is a floor, and as of five minutes ago, I can now adjust it on the fly. See below, in the "Tweaking the Numbers" section.

  4. Make stories wait 24 hours

    The maximum votes should provide this wait time. If it's not, it should be raised. (Again, see below).

  5. Change the thresholds

    Now we're getting somewhere. When something isn't working to everyone's satisfaction, there's always a rush to invent something totally new and different. I think not enough attention has been paid to the idea of simply adjusting all the knobs on what we have for a better result. There are a bunch of knobs, and if we give some thought to what's causing the dissatisfaction, we may be able to work out the right balance.

  6. Rate stories as comments are rated

    Statistically, I think this is a better idea than voting, for the same reasons our averaging comment rating is better than Slashdot's additive style. Socially, additive voting definitely has advantages of clarity and immediacy. It would be interesting to try a story-rating system, but like I've said, I have limited time and many other things to do. I'm sorry to say, a wholesale overhaul of voting is not even in my top ten at the moment. If anyone wants to try coding it, please do so with my utmost blessing. :-)

  7. and
  8. do not require comment, I think. :-)

Tweaking the Numbers

I think the first thing that ought to be tried is a close look at what the actual parameters are, and how they might be adjusted. What I'm lacking so far is a clear sense of what the problem is, from people objecting to this system. Is it posting stories with too little general interest? Stories that are too low-scoring in the straight votes? Posting things too fast? All of the above? More information on exactly what's wrong with the things being posted would be very helpful.

But let me describe things that can be adjusted in the current scheme:

  • Normal posting thresholds: Not sure if this would make any difference, but obviously post and dump thresholds can be raised or lowered. In fact, the posting threshold was raised recently, to 95 from 80. Thy can also be user percentages, but as discussed elsewhere, that isn't working too well at the moment. For that to really work properly, we need to have some sense of "active users" I think.
  • Maximum votes: Right now, it's 350. The intention was to have this keep stories around for a day or two, and give everyone a chance to at least see them before making a decision. This could be raised, and sounds like it probably ought to be.
  • Auto_post_section and auto_post_front_page: These are the thresholds for auto-posting. They are arbitrary precision numbers between 1 and 5 inclusive. The story's overall (average of vote and comment) score is comapared to these to determine whether to post or not. Currently, they are 3.35 and 4.2 respectively.
  • auto_post_floor: The minimum score to consider for auto-posting. If a story's voting score is lower than this, when it reaches max_votes, it will just be dropped. Currently this is 5 (changed from 0 in the last half hour).
  • auto_post_min_ratings: The minimum number of comment ratings to consider. I recently looked at the code, and it turned out I had misremembered how the comment score worked. It does this:
    • For each rated topical comment in the story, get the overall comment score, and the number of rating data points contributing to that score.
    • Weight the ratings by the number of points contributing to them. A comment with a score of 3.5 as a result of 5 ratings will act like 5 comments with a rating of 3.5. A comment with a rating of 5 as a result of a single rating will be treated as one comment with a score of 5.
    • Average the weighted ratings. Use auto_post_min_ratings as the "ratings floor". That is, in the example above, we'd do ((5 * 3.5) + (1 * 5)) / 6, provided 6 was greater than min_ratings. Right now, min_ratings is 20, so it would actually calculate ((5 * 3.5) + (1 * 5)) / 20, acting as though there are 20 rating points on the story. This provides a penalty for stories that do not have much ineresting discussion going on, in theory, while being resistant to one person trying to "skew" the result by rating everything 5.

    Hopefully that's clear enough for people who can't or won't look at the code. The minimum number of comment ratings can be raised or lowered to change how much discussion is considered "keep it"-worthy.

So that's the system we've got. These are things that can very easily be changed. So what exactly is wrong with the stories that are being posted and making you unhappy? Maybe we can see how the knobs can be twiddled to solve the problem without a whole lot of re-engineering.

____
Not the real rusty
A small point ... (5.00 / 3) (#40)
by Ranieri on Mon Feb 25, 2002 at 11:39:06 AM EST

Maximum votes: Right now, it's 350. The intention was to have this keep stories around for a day or two, and give everyone a chance to at least see them before making a decision. This could be raised, and sounds like it probably ought to be.

Apparently the aim is to make stories stick around for one or two days. The main problem of the system as it now stands is that this is not happening. As i see it we now have two options: we can turn this number up and until the average queue permanence time is around 36-48 hours, or we can wait 36-48 hours and then feed the story to the autoposter simply based on time.
The advantage of the latter method is that, no matter how much the site grows, it will always stick around for 36-48 hours unless voted up or down.
Of course, simply modifying the threshold requires little code changes, and is not a bad first line of defense regardless of the final decision.
--
Taste cold steel, feeble cannon restraint rope!
[ Parent ]

Posting too fast (4.50 / 2) (#42)
by wiredog on Mon Feb 25, 2002 at 11:43:23 AM EST

It leaves insufficient time to read the stories and the comments. Lately, I tend not to read most of the comments due to a lack of time. Another problem is duplicate stories. Especially with all the newbies around here. There was one that was submitted last week that had the same content as one that had been posted two or three days previously. It had several "+1 FP" votes before some of us said "waitaminute, this is a dupe!". Having a story hang around for more than a couple hours would be a good thing.

I don't think we want a "min time in queue" variable. Some stories get 90%+ "+1" votes in a short period of time, which indicates strong and widespread interest in the story (that seems to be what happened to this one).

Last year I had a few stories that hung around in the queue for several days, until I asked the editors to drop them. In general, I would rather have a story whose score hovers below post and above drop stay in the queue for three days, than have it get posted in 6 hours because of comment volume.

Two things I'd like to see. One would be a checkbox on the submission form for "Post to my diary if this gets voted down", which would have the benefit of saving the context of interesting discussions. The other would be a "max time in the queue" variable. Maybe a drop down list with values of, say, 12 and 18 hours and then one day up to a week, incrementing by days.

Peoples Front To Reunite Gondwanaland: "Stop the Laurasian Separatist Movement!"
[ Parent ]

Voting can get devalued (3.66 / 3) (#44)
by UncleMikey on Mon Feb 25, 2002 at 12:19:13 PM EST

My current complaint (because I just got bit by it) is that a complaint that's on its way up, e.g. near the posting threshold and climbing, can still find itself dumped. The Danny Pearl story had a vote of +89 when it got dumped after hitting the 350 threshold, presumably because the topical comments associated with it were all relatively low rated. Now, there were other problems with that story, and perhaps it didn't deserve to have reached +89 in the first place, but my point is, having reached +89, to get dumped seemed extremely strange.
--
[ Uncle Mikey | Radio Free Tomorrow ]
[ Parent ]
Grrrr... (none / 0) (#48)
by UncleMikey on Mon Feb 25, 2002 at 12:42:54 PM EST

s/a complaint that's on its way up/an article that's on its way up/

I need more coffee.

[ Parent ]

More data - currently about 10 hours to post? (5.00 / 1) (#46)
by TheophileEscargot on Mon Feb 25, 2002 at 12:24:20 PM EST

I've gone back to my original data, and for the nine stories posted through Maximum Votes, and recorded the number of hours between the first comment and the story being posted to the front page / section.

The mean time was 8.2 hours. The individual times were 7, 5, 4, 10, 7, 24, 4, 7, 6 (I only took it to the nearest hour)

Plucking a figure out of the air, assume it takes two hours to post the first comment (I think the real figure would probably be lower). That means it's taking about 10 hours on average for a story to get posted through Maximum Votes, so if the target is 36 - 48 hours, the threshold needs to be significantly increased.

This sort of thing could really do with a script to get accurate figures, if there's anyone feeling clever out there ;-)
----
Support the nascent Mad Open Science movement... when we talk about "hundreds of eyeballs," we really mean it. Lagged2Death
[ Parent ]

If I was given the admin passwords... (4.00 / 1) (#49)
by TheophileEscargot on Mon Feb 25, 2002 at 12:48:35 PM EST

...I'd set the thresholds as follows ;-)

Normal posting thresholds: fine as they are (95 post, -20 dump).

Maximum votes: 1200. This sounds like a lot, but if you see my other comment it's currently taking about 10 hours to post, when it ought to be 36 - 48. This change is actually slightly less than would be needed to achieve that.

Auto_post_floor: 48. If you can't get half the post threshold, I think you ought to resubmit.

Auto_post_front_page: 5. I don't think a story should get to the front page by this route.

Auto_post_section:...well I'd be tempted to set this to 5 as well, thus neatly dumping everything! But given the whining and bitching I'd get from ungrateful users, under protest I'd make it 4. This is somewhat more difficult than now to account for inflation, but not unreachable.

auto_post_min_ratings: This makes my brain hurt. I haven't a clue.

In addition I'd display the maximum votes threshold with the post and drop thresholds. I think that one thing that bugs people is the lack of transparency: they're not sure exactly what's going on.
----
Support the nascent Mad Open Science movement... when we talk about "hundreds of eyeballs," we really mean it. Lagged2Death
[ Parent ]

Ok, changed to time (5.00 / 2) (#51)
by rusty on Mon Feb 25, 2002 at 01:01:42 PM EST

Alright. bowing to what seems like good sense, and a better fulfillment of the original idea, I've changed it so that the auto-post is now time-triggered instead of vote-triggered. That is, stories that don't hit a hard threshold will now hang for a maximum of 36 hours (as currently set) regardless of number of votes. This does ultimately seem to make more sense than continuously increasing the number of votes to adjust for number of users.

One caveat though is that this whole system is only triggered by a vote. So if everyone has already voted after 24 hours, we're screwed. :-)

I don't think that's gonna be a problem though. We'll see how this works...

____
Not the real rusty
[ Parent ]

Brilliant! (3.00 / 2) (#55)
by TheophileEscargot on Mon Feb 25, 2002 at 01:21:28 PM EST

This should work, and if it doesn't, localroger and Ranieri can be blamed, plus the fact that it was the most popular poll option. ;-)

I think one bonus of this is that it will give "marginal" posters an incentive to brush up the quality of their stories, since that way they get instant gratification instead of hanging around for 36 hours.

Also, it will look less "weird" to people, since a greater proportion of stories hit the post threshold.

I promise I'll hold off on the Meta-stories for a while now. At least a week ;-)
----
Support the nascent Mad Open Science movement... when we talk about "hundreds of eyeballs," we really mean it. Lagged2Death
[ Parent ]

Blame me! (5.00 / 2) (#72)
by Ranieri on Tue Feb 26, 2002 at 04:19:31 AM EST

This should work, and if it doesn't, localroger and Ranieri can be blamed

I'll gladly take my part of the blame if localroger does the same :)
--
Taste cold steel, feeble cannon restraint rope!
[ Parent ]

"You the man." [NT] (2.50 / 2) (#57)
by THoliC on Mon Feb 25, 2002 at 01:57:47 PM EST




"Wanderlust,
has got us both,
looking for a bed today..."

[ Parent ]
Beelzebozo! (2.00 / 2) (#58)
by kaemaril on Mon Feb 25, 2002 at 02:02:22 PM EST

"...It's not rocket science; I can prove it with an etch-a-sketch..."

God, I miss Bill Hicks. He died before his time. Can you imagine what Bill would make of Bush and Ashcroft?


Why, yes, I am being sarcastic. Why do you ask?


[ Parent ]
Amen brother. (3.00 / 3) (#59)
by THoliC on Mon Feb 25, 2002 at 02:09:05 PM EST

We need him (or someone like him) more than ever.

T.


"Wanderlust,
has got us both,
looking for a bed today..."

[ Parent ]
You forgot the last option! (4.00 / 3) (#52)
by FredBloggs on Mon Feb 25, 2002 at 01:04:23 PM EST

9. Devote an entire site to discussing moderation, voting, karma eta. (Sort of like Ham radio, where all you`re allowed to talk about is your location, swr readings and the size of your `twig`.)

Seriously, who gives a shit! Replace the submission queue with a `i dont care` check box - instead of voting the story up or dumping it, just tick the box and you`ll never have to see it again.


[ Parent ]
Humble request (4.00 / 2) (#60)
by jabber on Mon Feb 25, 2002 at 02:33:47 PM EST

Regardless of how you ultimatelly decide to manage the posting of stories, would it be possible to denote in the story header the means by which stories were posted?

For example, adding an icon to the story header that would indicate whether the story was overwhelmingly voted up, or hit a time barrier, or a vote barrier.. Maybe include the for/against vote ratio and Time In Queue (spent/max) ratio??

It would solve no problem to provide this information, and for most people it would probably be extrenuous, but I think it'd be interesting to see that data.. Do it in the name of 'glasnost' :). Along-side the retained voting record, it would add value for anyone interested in the 'merit' of the story..

[TINK5C] |"Is K5 my kapusta intellectual teddy bear?"| "Yes"
[ Parent ]

central editor or distributed editorships? (3.00 / 1) (#62)
by ndw on Mon Feb 25, 2002 at 04:24:35 PM EST

2.Abolish the Maximum Votes threshold

Not gonna happen. Well, to generalize, we will not return to an open-ended system. If many of you either weren't here or don't remember it, it sucked. The complaints were far more vocal and constant then they are about this.

Maybe K5's system of handling votes/stories is too centralized or in other words relies too much on too few localized editors like rusty, etc.

Maybe K5 needs to evaluate their system and move to a distributed environment that safely and cleanly accepts an increased number of editors with the trust metrics established under the central editor theme rusty has going now. (ownership, operations and permissions to the right people to do pre-approved or commissioned things)

This way, the complaints can be handled as trouble tickets by more than a few central people (20-30). They can be handled by a crew of 100's.

Isn't this approach more open-ended?

[ Parent ]

I don't think that's a good idea (4.00 / 1) (#64)
by rusty on Mon Feb 25, 2002 at 04:43:38 PM EST

What we want to happen is to remove editors from the process. Why is one person choosing based on their opinion better than the system choosing based on information provided by many many users? Any solution that requires more editors is likely a bad solution from the perspective of the intent of the whole site.

____
Not the real rusty
[ Parent ]
He *said* that? (2.00 / 1) (#70)
by kmself on Mon Feb 25, 2002 at 06:15:11 PM EST

"Imagine how boring the world would be if all comment systems were as flawless as K5s?" --Rob Malda

Damn, I'm good ;-)

Unfortunately, the comment appears to have disappeared from /.'s database.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Yup (2.00 / 1) (#71)
by rusty on Mon Feb 25, 2002 at 06:42:54 PM EST

I promise he actually wrote it. The whole discussion has since been "reaped" to oblivion by the wonderful /. archiver. I've left the quote, and the link, for irony value. :-)

And yes, he was being sarcastic. As far as I could tell.

____
Not the real rusty
[ Parent ]

Don't approximate the solution... (5.00 / 5) (#41)
by chipuni on Mon Feb 25, 2002 at 11:39:27 AM EST

The goal for the maximum votes policy was to make sure that articles don't stay in the queue forever. The goal was to leave stories in the queue for up to 48-72 hours to give them the chance to be voted completely up or down.

If that's the case, then implement that, rather than setting a maximum number of votes.

I recommend that we eliminate the maximum votes threshold completely, and instead implement a maximum time threshold. If a story has been in the queue for, say, 72 hours, and it has neither reached the maximum nor the minimum number of votes, then it should be chosen or dropped based on a second algorithm (like what we have now, based on the maximum vote threshold.)

This will give people of all time zones the chance to vote on questionable stories. This will scale, no matter how much Kuro5hin grows or shrinks. And it is equally fair to the most controversial stories as the current system.
--
Perfection is not reached when nothing more can be added, but only when nothing more can be taken away.
Wisdom for short attention spans.

Done (4.00 / 3) (#56)
by rusty on Mon Feb 25, 2002 at 01:44:04 PM EST

I agree. It's 36 hours right now. We'll see if that helps.

____
Not the real rusty
[ Parent ]
Do not put 36 hours arbitrary value! (2.00 / 1) (#77)
by johwsun on Wed Feb 27, 2002 at 08:39:30 AM EST

Let me disagree with you.
You should better cast a permanent ballot between 0 and 1 where 0 is the oldest message in the queue and 1 is the newest message.

Believe me, believe it! People ARE responsible, they will not wast your system by voting stupid values.

[ Parent ]
Tweaking, and one additional point (2.00 / 2) (#43)
by TON on Mon Feb 25, 2002 at 12:18:02 PM EST

Tweaking

Please raise the maximum votes threshold. A hard time limit seems a bit to restrictive, the limit needs to come up in step with the change from 80 to 95.

Also, maybe look at raising the Auto_post_section and auto_post_front_page from their current 3.35 and 4.2. I don't know what the values should be, but maybe just try boosting them a wee bit, especially Auto_post_section.

Additional point

Is there some way to incorporate the ratings of editorial comments into the autoposting? A highly rated ed comment, like this one for example, often has great suggestions for how to improve the story for resubmission. Weighing highly rated ed comments in an inverse way would seem to address story quality more directly. Also, this might encourage more ed comments; not a bad thing IMHO. I know this is more labor, but it's offered for consideration.

"First, I am born. Then, the trouble begins." -- Schizopolis

Ted


A radical proposal. (4.25 / 4) (#47)
by i on Mon Feb 25, 2002 at 12:34:19 PM EST

First, implement the 72 hour (or whatever time is considered OK) delay, Then get rid of the "Vote" button altogether.

This way people would vote the article up or down solely based on quality of discussion it generates. Hmm...controversial. Maybe implement this for just one section, as an experiment?

and we have a contradicton according to our assumptions and the factor theorem

Don't you mean... ? (4.00 / 2) (#61)
by boxed on Mon Feb 25, 2002 at 04:12:42 PM EST

Don't you mean: "x hours of delay without having access to the vote button, then add it."?

[ Parent ]
Yes (5.00 / 1) (#89)
by 5s for Everyone on Thu Dec 19, 2002 at 08:18:08 AM EST

I believe he did. Getting rid of the "Vote" button would be pretty useless, don't you agree?
--
There is Damezumari in the Bamboo Joint
[ Parent ]
No, (none / 0) (#90)
by TheophileEscargot on Sat Jan 04, 2003 at 06:49:59 PM EST

I'm pretty sure he didn't. I think the idea was that after 72 hours, the decision would be made on the average rating of the topical comments. (At present, that is only part of the algorithm).
----
Support the nascent Mad Open Science movement... when we talk about "hundreds of eyeballs," we really mean it. Lagged2Death
[ Parent ]
Why voting is a bad thing (3.33 / 3) (#63)
by SIGFPE on Mon Feb 25, 2002 at 04:35:06 PM EST

Voting is inherently a kind of bottleneck. The tastes and interests of all the different people who use K5 are all funnelled through a single system to obtain a global score for every submission. This is just a hangover from the old days when resources were more limited and there always had to be one winner. But in a forum like K5 we don't need to eliminate bad stories if instead we each have our own personalised ratings system and we select the stories we want to see in our own view. Voting to select what goes into someone else's view is a bit like living in a society where we get to vote on what other people get to do in their own bedrooms. (Oh...we do...oh well, most of us wish we didn'.)

What I propose is that we each individually rate whatever comments and stories we want. Then any one of a set of standard algorithms is used to extrapolate and predict what articles would be of interest to each of us (a bit like the amazon.com ratings system). One method might be to use something sophisticated like PCA and skim off the top eigenvalues though there are simpler schemes too. From such an analysis you could still extract a 'generic' global score prediction that could be used to choose what non-registered users would see, but once you register you'd get to eventually completely determine the K5 that you see. Far more democratic IMHO.
SIGFPE

Community vs. Individual (4.00 / 3) (#65)
by rusty on Mon Feb 25, 2002 at 04:52:21 PM EST

That utterly fails to appeal to me, because K5 is intended to be a collaboratively produced site by a community of participants. I know there are various ways we can make everything personalized and individual, it just holds no interest for me. The interesting thing about K5 is that we're all deciding together. If we each got to determine the K5 we see for ourselves, we might as well just all go sign up with Blogger.

Far more democratic IMHO.

That's not democracy, it's individualism at best, and I think solipsism would be a better description. The web is already lonely enough, I don't want to help make it lonlier. :-)

____
Not the real rusty
[ Parent ]

Nah! (3.50 / 2) (#67)
by SIGFPE on Mon Feb 25, 2002 at 05:26:38 PM EST

This isn't 'individualism'. The point is that a prediction about your own interests is made based on others who share an interest in you. With something like PCA you have a whole continuum going from a global score shared by anyone to a score that's so specific it has no predictive value because it can only predict what you've already done. Somewhere in between you can compress the space of people's tastes down to an n-dimensional subspace corresponding to n types of taste. If you make n small you find that K5 will split into broad classes (like culture, technology etc.) like you already have (except reflecting people's real interests rather than an artifically determined one) and if you make n large it fragments into small individual interest groiups. The owner of the site can take their pick.
SIGFPE
[ Parent ]
Maximum Votes threshold arbitrary (2.00 / 1) (#73)
by johwsun on Tue Feb 26, 2002 at 08:46:54 AM EST

>Dump everything at the Maximum Votes threshold
>This is the solution that I now prefer.

I agree with you, but I dont agree with the fact that someone has to set <Maximum Votes threshold> to a prefix number. This is arbitrary, so it is wrong.

I propose a permanent ballot where users will be able to vote a number between 0 and 1 , where 0 means <Maximum Votes threshold = 0%> and 1 means
<Maximum Votes threshold = 100%>.
The final <Maximum Votes threshold> value will be the average of all people's votes.

Do you agree with that ?
I am proposing similiar votes for all arbitrary values of the system.

I disagree (4.00 / 1) (#74)
by hurstdog on Tue Feb 26, 2002 at 02:45:46 PM EST

Right now there are approximately 104 variables in scoop. About 25+ of those are "arbitrary values" which we use to configure how the site works. Allowing people to vote on these would require a few changes and would not work well, in my opinion.

First, we'd have to have regular polls on the values, or running polls on them. Running polls would require some code changes, I think. Also, this would mean everyone would have to know how scoop works, to be able to vote well. Add to that needing to know specific stats for stories posted, dropped, etc, and it becomes too much of a burden for casual readers. I think site configuration type things are too specific and should not be run by consensus.



[ Parent ]
So... you dont disagree...? (none / 0) (#75)
by johwsun on Wed Feb 27, 2002 at 02:23:48 AM EST

I want to extend my previous proposition to all the arbitrary algorithms you use, for example the algorith that is used to calculate the score in order for an article to appear in the fron page.
I believe that all those arbitrary algorithm should be a subject of a permanent vote.

You do not tottaly disagree with me, do you?
You just said it is just difficult to code regular ballots which affect running code, this does not mean that you tottaly disagree with the idea. I am also a programmer. I understand what you have done here and how much difficult is this I am proposing (it is not too much is it?)

>Right now there are approximately 104 variables >in scoop. About 25+ of those are "arbitrary >values"

Why not vote for the rest 79 if you dont mind?


>Also, this would mean everyone would have to >know how scoop works, to be able to vote well. >Add to that needing to know specific stats for >stories posted, dropped, etc, and it becomes too >much of a burden for casual readers. I think >site configuration type things are too specific >and should not be run by consensus.

This is what exactly argument your government use when you criticize a law they created, or when you criticize the arbitrary values they put to you when they decide about the taxes you pay.

Why not try something different here?
Just try it, then drop it if you dont mind!



[ Parent ]
I ll tell you a story... (none / 0) (#76)
by johwsun on Wed Feb 27, 2002 at 03:22:19 AM EST

Back in the 1994 there was a site named hypernews.
It still exist at www.hypernews.org , but is a dead site now. It was the first one that proposed threaded discussion and it was very inovative for that time and very popular.
I have done many debates there about the need of "reader's ratings". They did not agree with me, the just said "ratings didnt rate". Now the site is dead, but readers ratings and poll are alive as you cas see everywhere.
I am not a very good programmer, but i tried to create a clone of hypernews at http://egnatia.ee.auth.gr/HyperNews/get/eforum.html in order to add readers ratings. I failed due to lack of popularity and programming skills.

But i think I can make another prediction now:

I you dont incorporate in your code permanent ballots for arbitrary values and algorithms, in order for readers to vote what they desire and be able to change their vote whenever they wish, someone else is going to do it and take all the popularity you have right now.

am I wrong? we will see....

[ Parent ]
No way (5.00 / 1) (#79)
by hurstdog on Thu Feb 28, 2002 at 11:13:00 AM EST

If you dont incorporate in your code permanent ballots for arbitrary values and algorithms, in order for readers to vote what they desire and be able to change their vote whenever they wish, someone else is going to do it and take all the popularity you have right now.

So if a site doesn't have permanant polls, then it will die? Well, guess I better stop visiting slashdot, cnn, yahoo.com... I really doubt that people decide what site to view depending on whether or not they have perpetual polls. We're doing fine without them now, steadily growing at about 150,000 hits per day. Slashdot is well over 2million hits a day, and they don't seem to be decreasing (ignoring the "Slashdot is dying!" folks).

Our site is not based around polls. If it was, then I would probably have already coded up perpetual polls. But people come here for the discussion, the polls are like a little bonus.

Overall, if people leave kuro5hin because we lack this feature, then we have more problems then just that lack of a feature.



[ Parent ]
permanent polls and information flow. (none / 0) (#82)
by johwsun on Fri Mar 01, 2002 at 02:20:49 AM EST

Permanent polls about algorithms and arbitrary values will make information flow better. Better flow of information will attract more readers, so more popularity.
And yes, I also think that slashdot is dying, and maybe you are going to kill it.
Dont mention cnn or yahoo. This is the other side of the coin, so there is no competition with them.

[ Parent ]
Best of luck to ya in the in the upcoming play (none / 0) (#80)
by Justinfinity on Thu Feb 28, 2002 at 12:25:31 PM EST

Good luck stealing the poularity :-) no, really. but don't steal too much, rusty still needs _some_ income ;-)

You seem to have vision, but you lack drive.

-Justin
a toast to the host who can boast the most roast

[ Parent ]
I am lazy... (none / 0) (#83)
by johwsun on Fri Mar 01, 2002 at 07:08:58 AM EST

I dont want to steal anyone's popularity. I am too lazy to start anything. But I am very glad to see my ideas come truth and very disapointed when I cannot see them realized somewhere. Thats why I am very glad about k5.

You got the point: I have vision, but I lack drive... I have more ideas, and I am waiting for someone to realize them ( and win maybe..)


[ Parent ]
you misunderstand (none / 0) (#78)
by hurstdog on Thu Feb 28, 2002 at 11:05:39 AM EST

You just said it is just difficult to code regular ballots which affect running code, this does not mean that you tottaly disagree with the idea

I didn't say they'd be difficult, they'd just require a few changes. I meant it as its something that we can't start using immediately, since I'd have to change a few things.

Why not vote for the rest 79 if you dont mind?

The other 79 or so are all configuration vars that you can't set w/out changing k5 completely. Stuff like the local email for the site admin, whether or not to cache, the site name, url, sendmail program, etc etc. Some of these are on/off switches to turn on ads, mojo, and other things. The list goes on.

I didn't mean to say there are only 25 that you need to really know how they work to change. Download Scoop, see how long it takes you to understand what everything does. Then see if you think a casual site reader can understand how to change these settings effectively. Its too much work for the regular site reader. It would take so much work for people to catch up and maintain, I postulate that it would become a site about running k5, not about discussion anymore. We'd be overrun with discussions about whether mojo_min_trusted should be 18 or 8, and how other variables are set.

This is what exactly argument your government use when you criticize a law they created, or when you criticize the arbitrary values they put to you when they decide about the taxes you pay.

It is. And for a lot of the things they say that about I agree. I do not by any stretch of the imagination understand tax law well enough to help them make decisions on it. There are very few things I understand that well, that I would trust myself to help make laws about it. Probably perl programming would be one of the few ;-)

Why not try something different here? Just try it, then drop it if you dont mind!

I'll bring it up to rusty, but in my opinion its not worth the effort to give it a shot. I don't think its something that would work. We will talk it over though.



[ Parent ]
Searching for the prefect number (2.00 / 1) (#81)
by johwsun on Fri Mar 01, 2002 at 01:58:19 AM EST

I postulate that it would become a site about running k5, not about discussion anymore. We'd be overrun with discussions about whether mojo_min_trusted should be 18 or 8, and how other variables are set.
I did not said about discussions. None will discuss. Everyone will be just able to vote the number he thinks it is the correct. Look. Many algorithms exist about how you can make information to flow. And those algorithms may have or not have some arbitrary values. Nobody knows for sure if those algortithms are the right ones to use, or their arbitrary values are correctly initialized. I believe in the "try and error" method, so permanent ballots are the solution in order to choose new or older algorithms that fit our current need.

It is. And for a lot of the things they say that about I agree. I do not by any stretch of the imagination understand tax law well enough to help them make decisions on it. There are very few things I understand that well, that I would trust myself to help make laws about it. Probably perl programming would be one of the few ;-)
I may be more ignorant and more lazy than you are in order to understand how taxes work. But I believe in statistics. and I beleive that if everyone puts a value about how he thinks taxes should be, then the result will be the best one, and no "brillian" mind or governemt official can ever think about this correct number.

[ Parent ]
I don't understand (5.00 / 1) (#84)
by hurstdog on Fri Mar 01, 2002 at 01:48:00 PM EST

What you mean by information flow.? What do you mean for the information to flow? And how will that help us?

Everyone will be just able to vote the number he thinks it is the correct.

I think that that might work in theory, but not in practice. I'll repeat my argument, with a different example. Get 20,000 random people together, and let them vote on the specifications for the metal beams, support structures, etc for a large buiding. Then get 1 mechanical engineer, and let him choose the settings. Which building do you think would be more stable? I'm willing to bet the one by the mechanical engineer, by a long shot. If you think the one built by consensus would be more stable, why?



[ Parent ]
Information is not a bulding, it is a cloud... (none / 0) (#85)
by johwsun on Mon Mar 04, 2002 at 02:30:41 AM EST

let them vote on the specifications for the metal beams, support structures, etc for a large buiding.
I believe that both society organization and information organization resembles more to a cloud than to a building. There is no specifications, support structures and such things. I dont think that the building is the right model to use in our case.

What you mean by information flow.? What do you mean for the information to flow? And how will that help us?
Many pieces of information may arrive to a system, and the way we accept some of them and refuse some others, defines the model we use in order to classify the information. I believe that "Readers Ratings" with the combination of some algorithms (like mojo) rated from the readers is the best model that can be used in order to make information to flow best.
We can vote for algorithms, decisions e.t.c., but how can we vote for numbers? I also believe that for almost every number we use in order to initialize an algorithm we can tally its maximum possible value to the number 1 and its minimum possible value to the number 0. By setting a value between 0 and 1 we can initialize the number. If all readers set their desired value between 0 and 1 , then the correct number for this algorithm will appear. Because I think that CORRECT IS CLOSER TO THE THING THAT MOST PEOPLE WANT.
Tell me the initial values you use, and I will tell you how you can establish a permanent ballot for them.

[ Parent ]
hehehe I am right once again! (none / 0) (#87)
by johwsun on Thu Mar 07, 2002 at 02:24:19 AM EST

See whats happening here?
By not having rates on algorithms and on initial numbers, some guys (including the administrator of this site maybe?) taking advance of those algorithms and initial values and manipulate articles in the "moderation" section.
Also there in so respect on the minorities here. By implementing my proposotion, majorities could boost themselves by voting for algorithms, but minorities could protect themselves by voting on the initial values.
K5 is not what I was thinking of. Its a pitty, but anyway you have done a good start here! The one who is going to copy it and enhance it is going to go further...
of course a good protocol on rating and a pure P2P network is also needed (by using an ANDOS like protocol with blind signatures )

good luck and keep trying...

[ Parent ]
Redefine Post and Hide Threshold! (none / 0) (#88)
by johwsun on Thu Mar 07, 2002 at 08:21:02 AM EST

Through a permanent ballot.
Post threshold should be a vote between 0 and 1.
0 corresponds to 0 points and 1 corresponds to (All users of kuro5hin * 1)
Hide threshold should be a vote between -1 and 0.
-1 corresponds to (All users of kuro5hin * -1) and 0 corresponds to 0 points.

I am not posting it as a story because those majority guys are going to eat it again. but I am waiting for the minority(?) who implemented the +1,-1 algorithm to be fair enouth to let minorities live!

[ Parent ]
A question to hustdog... (3.00 / 2) (#86)
by johwsun on Wed Mar 06, 2002 at 07:05:14 AM EST

Let me ask you something else.
I posted a proposition about user profiles in "moderate submissions" section just a few minutes before.

Then some guy named entersomething told me to buy some ads before posting stories. I replied to him that he is ignorant and some day he will implement my proposition, then my reply was gone! A few minutes later my whole story was gone. Is it what I am thinking? No free speech here?

thanks for the answer...

[ Parent ]
Maximum Votes Threshold Considered Harmful | 90 comments (85 topical, 5 editorial, 0 hidden)
Display: Sort:

kuro5hin.org

[XML]
All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!