Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

[P]
Submission voting trends

By SeanCier in News
Tue Jul 11, 2000 at 07:05:09 PM EST
Tags: Kuro5hin.org (all tags)
Kuro5hin.org

Recently I submitted a controversial story, and since the voting ran somewhat long as a result, I had a chance to notice a rather interesting trend in the voting. Specifically, the change over time of the 'current score' was correlated closely with the latest few comments on the article: that is, if the top one or two posts were negative, new votes tended to be -1, and vice-versa; the more opinionated the top post, the stronger the trend.


The sample size is 243 votes (thus far), and the trend seems to hold strongly -- of course, it would be possible to verify more closely by graphing actual voting stats. It's easy to see the implications -- people are influenced heavily by others' opinions.

A simple solution for Kuro5hin might be to not make discussion visible to those who have not voted (and hence formed their own initial opinion).

The Kuro5hin microcosm quite probably represents behaviour of larger systems as well -- eg, democratic policitical voting by a relatively uninformed populace. What do people think -- what's the meaning of this trend in the small and the large, and what solutions might be viable in both scopes?

Sponsors

Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure

Login

Related Links
o Kuro5hin
o Also by SeanCier


Display: Sort:
Submission voting trends | 45 comments (33 topical, 12 editorial, 0 hidden)
Kuro5hin used to have blind voting... (4.00 / 7) (#4)
by Anonymous Zero on Tue Jul 11, 2000 at 06:49:01 PM EST

...but I think the consensus was it was undesirable. Read Meta: Submission Queue and Story Moderation Issues . Quote rusty: Of glaring problems with Kuro5hin, the one bugging me the most is managing the submission queue. IMO it's arranged around the wrong model -- blind voting. Instead, it should be more of an editors' bull room. Story submissions aren't a popularity contest, they're a dialog, though at a meta-level. This isn't the time for feedback on the content of a piece, but on its packaging, relevance, and appropriateness.

...and I personally agree. The single most important feature I like about K5 is you are told WHY you're story is being rejected by the very audience that is rejecting it. Yes, some readers are the highly suggestible type, but humans are flawed. I like the system as it is now.

Re: Kuro5hin used to have blind voting... (3.50 / 6) (#5)
by eries on Tue Jul 11, 2000 at 06:51:48 PM EST

Actually, I thought one of the stronger features of the original system (along these lines) was that people's votes showed up right next to their comments - and since they could comment right in the same "submit" as voting, they were encouraged to put smaller, quicker comments like "not enough content" or "just MLP" or "troll" or whatever. Now, people have to go through a whole separate "Post Comments" link to post comments.

My suggestions (which I just thought of while writing this): bring back the old "comments" box for _EDITORIAL_ comments, but allow people also to post _TOPICAL_ comments via the post comments link.
Promoting open-source OO code reuse on the web: the Enzyme open-source project
[ Parent ]

Thats *my* quote -- and random comments. (3.00 / 1) (#14)
by kmself on Wed Jul 12, 2000 at 12:45:18 AM EST

Though Rusty posted the item mentioned above, the quote (and opinions) were mine. I'd posted the item to Scoop, where as any of 5,000 Anonymous Heros will tell you, these meta topics really belong ;-) Rusty referenced (and extensively quoted) the item here on K5.

Much of the moderation system at Kuro5hin is based on ideas I knocked around with Rusty while he was first putting the site together. The Sub queue has gotten better (and works pretty well, most of the time -- I tend to agree with it more than I don't), though it's obviously got some glaring bugs.

It would be very useful for newbies to take a look at the Scoop discussions, the What Now topic, and sid=moderation for some background on why K5 is the way it is -- and what might still make it better.

There is a list of unimplemented features, and there has been a lot of discussion about moderation, submission queue, filtering, and karma. The last tends to be a bit of a hot button.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

How about Blind Voting to start... (2.00 / 1) (#18)
by Anonymous Hero on Wed Jul 12, 2000 at 03:09:28 AM EST

With revisions allowed.

Then you'd see when people get swayed (more accurate stats) - as well as the discussion/feedback to authors (not just a bilnd yea/nay).

-- Ender, Duke_of_URL

[ Parent ]
Vote changing (none / 0) (#34)
by cesarb on Wed Jul 12, 2000 at 01:41:32 PM EST

I wonder if it would be better to be able to change vote later. Like it is possible in the comment ratings.

[ Parent ]
Another explanation... (4.00 / 5) (#6)
by nuntius on Tue Jul 11, 2000 at 06:53:49 PM EST

What if...
The voting trend doesn't follow the current posts, but rather
=>The latest posts are the opinions of the latest voters.

That's my belief. Check it yourself--the order of who posted a story in a thread almost exactly matches the order in which they voted.

Supposedly, stories are posted if the # of positive votes exceeds the negative by some percentage, according to an equation. Applying statistics to this situation (accounting for the large # of votes) shows that (at least for the case in question), the probability that your story will change percentage by several points is rather low. Its currently running a 6% lead. How far does it have to go???

Does anyone know the actual formula used so we can run some analyses with it? The FAQ mentions one, but doesn't give specifics...

I personally think we should come up with some mechanism for finalizing stories when the voting reaches something like a 98% solid position, as two stories in the queue probably already have. When a story's vote reaches a statistical standstill, what should happen to it? With even a small # of stories, this was bound to happen sooner or later.

Congrats to CeanSier and davidpfitz7 for posting possibly the most (non-)controversial stories in k5 history!! ;-)

Re: Another explanation... (2.00 / 1) (#7)
by nuntius on Tue Jul 11, 2000 at 06:57:29 PM EST

SeanCier and davidpfitz,
Oops.

Sorry to both of you for misspelling your nicks.

I can't believe how bad I was...
The 7 tacked on pfitz's nick came from his story's position at the time...

[ Parent ]
Re: Another explanation... (none / 0) (#15)
by SeanCier on Wed Jul 12, 2000 at 01:05:42 AM EST

> What if...
> The voting trend doesn't follow the current posts,
> but rather
> =>The latest posts are the opinions of the latest
> voters.

   Correlation is not causation; agreed. But nevertheless, most voters do not comment (only approximately 10% seem to do so), so we can observe the effect of a single comment on about 10 votes before it changes -- and additionally, the most effect the type of trend you describe could have would be a 10% correlation. For what it's worth, the trend I was commenting on did occur observably *between* posts; that is, it was possible to observe several voters being affected by a single post.

   A more quantitative analysis -- actually graphing the smegging thing with real stats -- would be able to easily filter that kind of feedback out, fortunately. (feedback feedback? hmm)

-spc

[ Parent ]
Re: Another explanation... (none / 0) (#35)
by hurstdog on Wed Jul 12, 2000 at 03:44:58 PM EST

Actually at the moment for a story to get posted it has to have 35 more positive votes than negative ones. i.e. the difference between the # of positive and negative votes must be +35. -Andrew

[ Parent ]
posts can be the suggestive (4.20 / 5) (#11)
by l4m3 on Tue Jul 11, 2000 at 09:06:35 PM EST

I tend to notice I will at times vote down a story that I would otherwise vote to post based on the first couple posts I can see. The most common reason for this is when the post is directed to the original article poster to add more content or a link or correct spelling/gammar.

It seems that few of the articles which get voted down in that manner are reposted so maybe a solution to that sort of voting would be a way to alert the author(possibly via the main page, not email), or a way for the author to add/modify their post after it has been inserted in the queue (thus satisfying the complaints).

another (quite simpiler option) is to by default hide the comments, and after you vote the comments appear and you are given a chance to comment on the article.

Unbiased is overrated (3.80 / 5) (#12)
by End on Tue Jul 11, 2000 at 11:23:20 PM EST

Well, that's the point. If there is something lacking in the story editorially (not in the opinion, but the other aspects of it) it's good to be able to post a comment and alert other readers to things they may have overlooked. I think it has tended to increase the quality of stories on the front page since voters became aware of others' perspectives on the story/subject.

The most common complaint of course is a lack of write-up in the article. Put it this way: Go ahead and post any screwball opinonated articles you want to, just do your research and explain yourself! Of course, if you don't have an opinion and are just linking to an interesting news story, please consider digging around for more on the topic, verifying some facts and coming up with some conclusions or interesting corollary.

-JD

Re: Unbiased is overrated (none / 0) (#27)
by slycer on Wed Jul 12, 2000 at 12:02:07 PM EST

I agree with this to an extent. I think editorial comments are fine, but currently we see both editorial and meat comments. (Or maybe not, my memory must be failing). Strip the meat comments - post the editorial..

[ Parent ]
Just for the heck of it.. (2.50 / 2) (#13)
by Inoshiro on Wed Jul 12, 2000 at 12:33:26 AM EST

I reviewed my votes on two stories that were in the queue (which I had previously voted on).. The first on was the "Internet Computing" one which, compared to the story also about distributed computing that had been posted to the main page, seemed a bit more advertisey.. Once I reached the bottom, I posted a comment stating this (and a few other questions) so that you'd know why I voted -1. I didn't really read any comments before I posted mine, but most seem to reflect my sentiment.

The other story was about big corporations and open source.. We've discussed this one a lot recently (or so it seems), so I tagged a don't care. I'd still vote "don't care" even though other people have put comments about why they voted -1 there -- because I don't feel too strongly one way or the other. I tend not to vote -1 unless the write up is really bad (using the word/content ratio, and other metrics), the story seems flame-ish, or the story is not useful (various spams). This wasn't the case, so I voted 0.

There are few times when I've read something and had a hard time deciding +1 or -1.. in such situations, I tend to vote 0 because if I don't have a strong enough opinion to know if a story is +1 or -1, then I'm content to let others decide (who will have stronger opinions). This stops me from being unfairly swayed. If the story was interesting, and I'd like to discuss, it's a +1. And you know why I'd vote -1 ;)



--
[ イノシロ ]
Blind Voting (3.30 / 3) (#17)
by FlinkDelDinky on Wed Jul 12, 2000 at 02:27:04 AM EST

I was always for blind voting and was disappointed when K5 switched to the open format. I haven't noticed any increase in quality on the front page. I do not vote as often. Although when I have voted I believe I've only been influenced by commenting once.

I just think you get a truer opinion in closed voting. I don't expect K5 to switch back though. The switch was a C change, K5 use to reach a consensus by the action of individuals. Now it's gone on to some bizzare group think collaboration system. I miss the pure democracy feel that K5 used to have.

Yet another meta story. (4.00 / 1) (#19)
by inspire on Wed Jul 12, 2000 at 04:21:00 AM EST

Actually, what would be ironic (in the incorrectish Morrisette sense) would be to submit a meta-story on the recent proliferation of meta-stories.

Sigh...
--
What is the helix?

Re: Yet another meta story. (none / 0) (#20)
by Inoshiro on Wed Jul 12, 2000 at 05:34:36 AM EST

Ahh, but this allows us to discuss not just the actual stories that appear, but also how the stories appear, etc. Some of this is decided by the benign admins (I am just one example, as I decided on the new polls today), but most of it should be decided by you.. This is your site (not "you" you, everyone who has an account + the AHs).

Yes, it can be annoying sometimes when someone makes a point of posting a meta story about something which you have firm opinions on, but some people can benefit from this. Although I'd say that perhaps a more real time discussion of it (so that things can happen faster) on #kuro5hin would be better, it's not easy to get all kinds of people from different timezones involved in a real time chat. Not to mention the people who don't IRC. Thus the meta stories.

You're right that a meta-meta story would push it too far, though ;-)



--
[ イノシロ ]
[ Parent ]
Re: Yet another meta story. (none / 0) (#33)
by cesarb on Wed Jul 12, 2000 at 01:39:47 PM EST

Do it. I think Id vote it up. And then we could discuss about an upcoming meta-meta-meta-story.

[ Parent ]
Re: Yet another meta story. (none / 0) (#39)
by malikcoates on Wed Jul 12, 2000 at 05:55:15 PM EST

Yes, but think about this. Calling things which are not ironic, ironic, is ironic!

[ Parent ]
Problems with the submission queue (3.00 / 1) (#21)
by fuzz on Wed Jul 12, 2000 at 08:32:19 AM EST

As I have stated in the past, I think that one of two things needs to happen:

1) No comments can be posted on a story before it reaches the front page. That would encourage people to vote +1 if they have something to say about a particular topic.

or

2) The comments posted to a story while in the submission queue would be editorial only, pertaining only to the story's relevance, formatting, and general appropriateness. These editorial comments would be deleted if/when the story moved to the front page to allow for content discussion.

Currently, there is no appreciable difference between a story being on the front page and simply rotting in the submission queue. The voting is a farce.

More interesting (1.00 / 1) (#22)
by savagegus on Wed Jul 12, 2000 at 08:44:42 AM EST

I have to say moderating on the topic takes some of the interest away. If instead articles were voted on. Then if they made it to the front page it would become (at least for me) a reason to view more on the article.
Another nice thought is that it would kill snap reactions. If people have to wait to be able to post a comment they might think the matter through some more and present a better informed opinion.
--- Too many freaks, Not enough circuses
Remove info to inform? (none / 0) (#23)
by duxup on Wed Jul 12, 2000 at 08:50:20 AM EST

This note confused me:
"The Kuro5hin microcosm quite probably represents behavior of larger systems as well -- eg, democratic policitical voting by a relatively uninformed populace."

So we should solve that by removing the related information provided by others, and just vote with one persons opinion? I don't see where the belief that people are ill informed supports the removal of information.


I like seeing other people's thoughts (none / 0) (#24)
by duxup on Wed Jul 12, 2000 at 08:52:11 AM EST

I think the ability to see the comments before I vote is good. Obviously the article wouldn't have been submitted if someone didn't feel that it should be posted. I think we should also be entitled to see people's feelings why it should not be posted or more support for it.

I don't know everything, and when some things come up that might note that information in the article is factually inaccurate that I did not know, I would want to know. I give both the articles and positive and negative posts equal grains of salt, if neither has sources they rarely influence my vote.

I've seen a few people note that they feel that they are influenced by the posts that for whatever reason are at the top and feel that is bad. I don't quite get that logic, because can't you just IGNORE it? Perhaps an option to turn it off would be good for those people.


Re: I like seeing other people's thoughts (none / 0) (#29)
by Anonymous Hero on Wed Jul 12, 2000 at 12:43:08 PM EST

I've seen a few people note that they feel that they are influenced by the posts that for whatever reason are at the top and feel that is bad. I don't quite get that logic, because can't you just IGNORE it? Perhaps an option to turn it off would be good for those people.

If you're talking about voting on the goodness of an article, that vote should be independent of previous posters opinions of it. Seeing other people's commentary before you vote tarnishes the independence of your vote. Maybe you can justifiably assert that your decisions are never swayed by the opinions of others, but I'm not willing to make that claim myself. Keep the comments to the discussion only, not to the voting, and you can more realistically say that your vote is independent of the opinions of others.

[ Parent ]

Re: I like seeing other people's thoughts (none / 0) (#42)
by duxup on Thu Jul 13, 2000 at 01:46:15 AM EST

I think that how I make decisions about "the goodness of an article" should be my decision. I prefer to get other peoples and opinions before making a decision. They may have ideas or information that I may have thought of to justify that the post should or shouldn't be up.

[ Parent ]
Am I missing something? (5.00 / 2) (#30)
by error 404 on Wed Jul 12, 2000 at 12:49:09 PM EST

The article seems to say that SeanCier has observed that comments can affect how people vote - that you can, by writing messages, affect the behavior of your readers.

Isn't that one of the basic assumptions of language? That what I write may have an impact on others?

It is interesting to observe and measure, but hardly surprising.

I prefer blind voting myself, but...
..................................
Electrical banana is bound to be the very next phase
- Donovan

Re: Am I missing something? (none / 0) (#41)
by ramses0 on Wed Jul 12, 2000 at 08:30:38 PM EST

You're exactly right, but the question is:

Should you be able to affect your readers before they have had the chance to vote, and voice their opinion?

<devil's advocate>If you can affect my vote, aren't you effectively stealing my vote?</devil's advocate>

--Robert
[ rate all comments , for great justice | sell.com ]
[ Parent ]

Lobbying (none / 0) (#44)
by error 404 on Thu Jul 13, 2000 at 11:42:27 AM EST

There isn't a right or wrong here, just different effects.

Myself, I prefer blind voting. It brings out the raw preferences of individuals. On the other hand, what we have now is more likely to bring about concensus.
..................................
Electrical banana is bound to be the very next phase
- Donovan

[ Parent ]

Only post comments after it has been voted (4.00 / 4) (#31)
by hubie on Wed Jul 12, 2000 at 01:29:48 PM EST

Why not withold the comments from view until after the article has been voted on in the same manner as how the vote tallies are witheld? This way you can form your own opinion (or not, with a zero vote), then join the discussion.

Place Vote button at bottom (3.00 / 2) (#36)
by dim on Wed Jul 12, 2000 at 04:07:48 PM EST

Maybe there's even a simpler solution to this: put the "Your Vote" combo and "Vote" button at the bottom of the page. This will at least force you to scroll to the end, and maybe you will pick up most of the comments on the way, and read them intently. ;-)
--
cat ~/.signature

straddling the fence (none / 0) (#37)
by djzoot on Wed Jul 12, 2000 at 05:26:43 PM EST

On one hand: It's true that displaying pre-vote commentary affects votes. During the discussion surrounding the change to our current format (from blind voting), I was of the pro-blind-voting train of thought. Regardless of people's dedication to the philosophy of productive-discussion == good article, folks still vote based on biases either personal (ie: MS == bad), or in desire to follow the herd (ie: everyone else thot this article sux ... i should lay low). I felt that the blind voting system reduced the herd effect, but ...

On the other hand: Displaying comments during the voting gives one an idea of some sorts of discussion that can come of the article's possible posting. Even if I think an article is useless fluff, but see that folks have already begun interesting threads of discussion, I am more likely to give it a positive vote ... because the discussion (not the article) is what k5 is about ... to me.

rob
--
There is no K5.

how about this as a compromise... (3.00 / 2) (#38)
by eries on Wed Jul 12, 2000 at 05:36:15 PM EST

When you get to the submissions Q, you see the following:

voting box + submit button
comment box
all non-topical (i.e. editorial) comments for this thread

Once you submit a vote, you can then see the topical comments, and post your own via the normal "Post A Comment" link.

Any comments you submit via "comment box" above are considered to be editorial comments, and should be treated just like comments from the old voting system. Maybe these comments should even disappear when the story goes to "press"

Just my $.02
Promoting open-source OO code reuse on the web: the Enzyme open-source project
Why not make it an option ? (none / 0) (#40)
by ejf on Wed Jul 12, 2000 at 08:01:02 PM EST

Really, think about it. People who like blind voting can vote blind, and people who would rather make a more informed decision could do so by reading the prior comments.

Its all a matter of personal preference, IMHO. If you think you vote better when youre not influenced by the posts, you should be able to do so. I personally prefer being able to read the comments beforehand -- there is always a chance that an article is completely bogus (as said, humans are flawed ;-)


--- men are reasoning, not reasonable animals.
I agree!! (none / 0) (#43)
by mrbob on Thu Jul 13, 2000 at 10:02:36 AM EST

So what makes a persons comment so helpfull when voting? When you go into a voting booth in november, you won't see a list of the previous voters choices and a comment. Why should K5's be different. By showing comments _before_ the vote, you are only biasing it, as SeanC has pointed out. If you want a truely accurate vote, lose the comments (untill after the vote), like the system _used_ to be. Of course, this puts the poster (sorry ;-) under a little more pressure to make a better quality post. Include as many links as you can, perhaps showing both sides of the "arguement." Then support your own decision. If youre just trying to enlighten the K5 crew to some Cool New Thing, then make your post stick out by making it sound as cool as you though it was. Bob

Submission voting trends | 45 comments (33 topical, 12 editorial, 0 hidden)
Display: Sort:

kuro5hin.org

[XML]
All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!