Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

[P]
Diaries' load on Scoop

By Potatoswatter in Meta
Fri Nov 17, 2000 at 01:43:58 PM EST
Tags: Scoop (all tags)
Scoop

I was just browsing through all article headers with the search page, out of curiosity. The introduction of diaries, I found, has increased the rate of stories being posted twentyfold.


Since 12/21/99 (auspicious date): 2091 stories.
Since 7/20/00 (in the middle, for a good comparison): 1213 stories.
Since 11/13/00 (diaries introduced: monday the 13th... Garfield wouldn't be pleased): 575 stories.

12/99..7/00: 878 stories, 5 months * 30 days = ~150 days; 5.853 stories per day.
7/00..11/00: 638 comments, ~113 days; ~5.646 stories/day.
11/13/00..~11:15 today: 575 stories, 4 days (beginning of 11/13..end of 11/16);

~143.75 articles per day! And because much traffic occurs in the after-hours, this is really a kinda low estimate.

This is really some increase in volume. Consider that every single diary entry has a poll, and every poll records the users that have voted in it. Wow!

And if that wasn't food for thought, consider that I'll probably post this to my diary if it gets democratically rejected!

Sponsors

Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure

Login

Poll
Are we gonna go to hell?
o Scoop/Intes.net can handle it. No problemo. 20%
o Rusty'll just cut the polls and it'll all be condensed. 4%
o I have no time to read this; I'm making a diary entry. 10%
o We must commit mass suicide to reduce diary traffic. 19%
o It'll be all taken away and we'll go back to the dark ages of 1337 hidden SID s. 9%
o Potatoswatter, you have no life and you should at least be doing homework. 36%

Votes: 154
Results | Other Polls

Related Links
o Also by Potatoswatter


Display: Sort:
Diaries' load on Scoop | 19 comments (17 topical, 2 editorial, 0 hidden)
You crack me up... (3.00 / 6) (#1)
by pb on Fri Nov 17, 2000 at 12:02:04 AM EST

The dark ages of 1337 hidden SIDs? It was what, like a whole 25 days before the introduction of Diaries that I found that?

Of course, it's possible that my favorite mis-feature might have sped up the introduction of diaries, but still...

I think they're very cool, and I don't see what would get bogged down. Searching? Storage? It can't be that bad, yet, I'm sure they'll figure out how to deal with it.

I'm just glad we've had more activity on the site; *that's* very cool. Think of it as an advertising expense. :)
---
"See what the drooling, ravening, flesh-eating hordes^W^W^W^WKuro5hin.org readers have to say."
-- pwhysall
Ehh... (3.50 / 2) (#4)
by Potatoswatter on Fri Nov 17, 2000 at 12:26:31 AM EST

Although the problem might not be bad now, when we're 5-10x bigger, who knows? My point (altho' I should have stated it... bad Potatoswatter! Edit your stories!) is that the traffic from diaries appears to dwarf all other activity on k5. This is an increase in raw activity, it's true, and that in itself is good.

Tho it does kinda screw the system up if people are more inclined to post (real) stories to their diaries, like Fluffy Grue did, and get it posted quick & dirty, than use the real moderation-queue system.

On the other hand, there's my usage of this ability to post unmoderated comments, to put things in for honest discussion that would fit nowhere else in k5 and really do need an organizing ToC for themselves (the main diary page, after I write more installments). I'm sooo happy that I now have this privilege.

IMHO the best solution would be to give people the ability to arbitrarily create new sections. Inventing this off the top of my ass, I'd say these sections wouldn't be listed on the main page but people would be able to "subscribe" to them, causing them to appear in that user's left nav bar and be listed in that user's info page. I would be able to create a Mute discussion; other people would be able to post articles to it. I could have a separate personal discussion, which would also be public? I decide access privileges? People would have to take the initiative to create their own diary. But I suppose the cat's already out of the bag... Whell it's all a big experiment.

myQuotient = myDividend/*myDivisorPtr; For multiple languages in the same function, see Upper/Mute in my diary! */;
[ Parent ]

dammit again. (2.00 / 2) (#6)
by Potatoswatter on Fri Nov 17, 2000 at 12:29:42 AM EST

Please tell me not to put this into a story submission now. (OK, I won't, but only because 1 story per night is enough...)

Well it'll go into my Master Scoop Story, which I swear I'll write up Very Soon. I really should go to sleep, my mind is blank and I feel a kind of fizzing sensation... do you get a numbing buzz from submitting stories?

myQuotient = myDividend/*myDivisorPtr; For multiple languages in the same function, see Upper/Mute in my diary! */;
[ Parent ]

Discussion Forums... (2.50 / 2) (#10)
by pb on Fri Nov 17, 2000 at 01:11:25 AM EST

Like "Yahoo Clubs"? That's an interesting idea, but likely to end up eating more bandwidth, too.

Maybe, in the future this will get implemented. Systems like Scoop really are what we have today instead of BBSes, and it'd be cool to see some of their less organized mechanisms for discussion get brought over into our world. Maybe we'll supplant USENET eventually, too! :)

In any case, I'd have a region devoted to posting filks (basically song parodies, in my case usually computer related ones...) and probably another one for talking about computer stuff (in my case, probably mostly C programming)
---
"See what the drooling, ravening, flesh-eating hordes^W^W^W^WKuro5hin.org readers have to say."
-- pwhysall
[ Parent ]
credit where credit is grue (2.66 / 3) (#2)
by Potatoswatter on Fri Nov 17, 2000 at 12:10:57 AM EST

Props to Fluffy Grue for his diary entry that was, if not the inspiration for my "research", the catalyst for me making a fuss.

Grue thinx that people should be more careful about putting content in their entries.

Will the problem (if it is one) lessen when the effect of everybody (a few thousand of us) saying "testing, testing" is over? All these things and more I should have put in the write-up.

myQuotient = myDividend/*myDivisorPtr; For multiple languages in the same function, see Upper/Mute in my diary! */;

Argh! (2.00 / 2) (#14)
by fluffy grue on Fri Nov 17, 2000 at 12:41:35 PM EST

I'm fine with whatever pronouns you want to use to describe me, but stop misspelling my name, damnit! :)

Also, I think you missed the point to that diary post (which was clarified in comments). I was sarcastically pointing out that while everyone else was spamming K5 by playing with this new feature, so would I. :)
--
"Is not a quine" is not a quine.
I have a master's degree in science!

[ Hug Your Trikuare ]
[ Parent ]

oh. (2.50 / 2) (#15)
by Potatoswatter on Fri Nov 17, 2000 at 02:04:44 PM EST

Arright, fluffy :v) .
I'm sorry, it was late, I kinda regret posting the story at all. Kinda stupid premise; Rusty obviously did the math to figure out what the extra load would be, and traffic will die down after everybody's made their first entry.

myQuotient = myDividend/*myDivisorPtr; For multiple languages in the same function, see Upper/Mute in my diary! */;
[ Parent ]
please don't take away our diaries (3.00 / 3) (#3)
by enterfornone on Fri Nov 17, 2000 at 12:14:37 AM EST

I think the biggest fear, is that the server will die under the load of people like me who are constantly hitting refresh to see if any more diary entries have been submitted.

Yes it's sad, but these diaries are my new addiction. I wouldn't know what to do with all the hours I would get back if this feature is removed.

--
efn 26/m/syd
Will sponsor new accounts for porn.
No Biggie (3.00 / 3) (#5)
by michaela on Fri Nov 17, 2000 at 12:28:23 AM EST

It's pretty much all database stuff with some templates for layout, isn't it? Disk space is cheap. Creative database indexing will go a long way. After that, archive archive archive.
--
That is all
should've put this in the writeup (repost) (3.25 / 4) (#7)
by Potatoswatter on Fri Nov 17, 2000 at 12:36:48 AM EST

This is a reply to comment #1. I think it's at least as interesting as the original post, which I kinda rushed on because the statistics were rapidly becoming out of date:

Although the problem might not be bad now, when we're 5-10x bigger, who knows? My point (altho' I should have stated it... bad Potatoswatter! Edit your stories!) is that the traffic from diaries appears to dwarf all other activity on k5. This is an increase in raw activity, it's true, and that in itself is good.

Tho it does kinda screw the system up if people are more inclined to post (real) stories to their diaries, like Fluffy Grue did, and get it posted quick & dirty, than use the real moderation-queue system.

On the other hand, there's my usage of this ability to post unmoderated stories, to put things in for honest discussion that would fit nowhere else in k5 and really do need an organizing ToC for themselves (the main diary page, after I write more installments). I'm sooo happy that I now have this privilege.

IMHO the best solution would be to give people the ability to arbitrarily create new sections. Inventing this off the top of my ass, I'd say these sections wouldn't be listed on the main page but people would be able to "subscribe" to them, causing them to appear in that user's left nav bar and be listed in that user's info page. I would be able to create a Mute discussion; other people would be able to post articles to it. I could have a separate personal discussion, which would also be public? I decide access privileges?
People would have to take the initiative to create their own diary, reducing the said low-S/N traffic. But I suppose the cat's already out of the bag...

Whell it's all a big experiment.

myQuotient = myDividend/*myDivisorPtr; For multiple languages in the same function, see Upper/Mute in my diary! */;

how about (3.50 / 2) (#8)
by enterfornone on Fri Nov 17, 2000 at 12:39:35 AM EST

Giving people the ability to vote to "promote" a diary entry into a "real story"? This would also solve the "if this gets voted down I'll post it to my diary" since all stories would orginally start life as diary entries.

Or is that just insane?

--
efn 26/m/syd
Will sponsor new accounts for porn.
[ Parent ]
Very cool. (3.50 / 2) (#9)
by Potatoswatter on Fri Nov 17, 2000 at 12:47:55 AM EST

I like that. However, diaries would rapidly become a wasteland for unknown and unsung stories.

The idea of giving submitters the option of having their stories in the submission queue put on their diary page or something, or being able to use your diary as a forum to edit & perfect your stories for true K5 worthiness before marking a revision up to submission status, makes me happy.

If only I'd used this as the topic of the fscking story! I want modification privileges! Just to append stuff on, pleeeze?...

myQuotient = myDividend/*myDivisorPtr; For multiple languages in the same function, see Upper/Mute in my diary! */;
[ Parent ]

Diaries should be personal (5.00 / 5) (#12)
by rusty on Fri Nov 17, 2000 at 02:41:10 AM EST

I don't really like the idea of allowing diary stuff to be promoted to the queue. First, you can do that already (this story started life as a diary entry, and was submitted as a story after several people lobbied Paul) simply by taking your text and posting it as a story. If you want to use it as an article "pre-release" queue, that's fine.

But I also want people to feel free to write any crap that's on their mind in their diary. It's *your* space. That's what it's for. To allow voting of diary stuff into the main content of the site feels like an invasion to me. At least, it would potentially make people more skittish about posting their random thoughts.

I'm not worried about the space that diaries take up. If I was, I probably wouldn't have done them, right? :-) If you're sorried about the server, here are some stats as of right now:

Current Stats:
  2:33am  up 63 days,  8:41,  1 user,  load average: 2.22, 2.08, 2.11
                       total       used       free     shared    buffers     cached
Mem:        971520     966216       5304     328964      20432     305732
-/+ buffers/cache:     640052     331468
Swap:       297152      18284     278868

Processes: 
  Apache:      61
  Mysql:       71

Open web connections:      38

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             486M  354M  107M  77% /
/dev/sda5             973M  458M  465M  50% /usr
/dev/sda6             486M   12M  449M   3% /var
/dev/sda7             201M  2.1M  188M   1% /tmp
/dev/sdb2              17G  1.1G   16G   6% /var/lib/mysql
/dev/sda8              15G  1.7G   13G  12% /var/lib/apache
Things to note: it's a dual processor box, and running dnetc, so the baseline CPU load is 2.00. If it's not doing *anything*, the load is 2. So, 2.22 is not exactly a crushing load. At peak times (5-7PM EST) it gets up around 2.6. Also not anything to be worried about.

Look at the last two partitions in the df output. sdb2 is only for mysql stuff, and sda8 is only for apache. As you can see, they each have plenty of room to grow. :-)

Another thing I think you're misinterpreting is the raw story numbers vs. the percentage of content. In their average length, diary entries are pretty comparable to comments. "Real" stories tend to be longer, and also draw a lot more comments. I doubt that diaries are taking up much more than 1/4 of the actual byte storage, despite seeming like a flood of stuff.

And a final point. I think fluffy's right, but that people will (and already have) start to calm down once the first week or so is over, and everyone's tested it out, and typed "First post!" and stuff. Give it a little time, before we start thinking about how it can be fixed. :-)

____
Not the real rusty
[ Parent ]

diary storys (none / 0) (#18)
by codemonkey_uk on Mon Nov 27, 2000 at 07:02:48 AM EST

it does kinda screw the system up if people are more inclined to post (real) stories to their diaries
One feature that I think would be benaficial is if diarys could be "nominated" to go into the story queue. Then, possably, voted into a section, or even the front page.


---
Thad
"The most savage controversies are those about matters as to which there is no good evidence either way." - Bertrand Russell
[ Parent ]

online diaries? (3.00 / 3) (#16)
by darthaya on Tue Nov 21, 2000 at 09:56:25 AM EST

I like writing diaries, to track my thoughts and also to see what silliness I had commited 1 week ago, 1 month ago, or maybe years ago. Is there other online diaries services that we can use other than k5?

re: online diaries? (3.00 / 1) (#17)
by bse on Thu Nov 23, 2000 at 10:01:27 AM EST

well i know of two.. blogger and diaryland. personally, i use blogger on my personal site.. as yet, i haven't ventured into using k5 diaries.. of course, taht could change. =D

---
"Please sir, tell me why, my life's so pitiful, but the future's so bright? When I look ahead, it burns my retinas." -- Pitchshifter - Please Sir
[ Parent ]

absolutely! (none / 0) (#19)
by CrayDrygu on Mon Nov 27, 2000 at 11:16:36 AM EST

I used Blogger for a long time on my website, and I have to say it's a great tool. Very easy to set up, and even easier to use.

Personally, though, I wanted something to run off my own site, rather than relying on an outside website to make my updates with, so I've switched to Greymatter. A little tougher to set up, but just as easy to use, and has more features (reader comments, "karma" for posts...all optional, of course).

If you need somewhere to host it, there are sites that offer both space to host your diary and a convenient way to update. Rather than link to just one or two, here's a link to Google's directory of Personal Diary Hosts.

Have fun!

[ Parent ]

Diaries' load on Scoop | 19 comments (17 topical, 2 editorial, 0 hidden)
Display: Sort:

kuro5hin.org

[XML]
All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!