Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

Well. That Sucked.

By rusty in Site News
Sat Dec 15, 2001 at 04:29:26 PM EST
Tags: Kuro5hin.org (all tags)

As you can see, K5 is finally back. I don't have very much to add to the ongoing history chronicled in our dispatches from the front lines of Operation Enduring Downtime, but I did want to profusely thank some people, and describe the new hardware setup both current and near-future. Update [2001-12-16 1:9:35 by hurstdog]: You might be seeing a few Internal Servers Errors when you load a page. Maybe about 1 in 20 loads. We know about it, no need to email rusty. I'm digging into it now.
Update [2001-12-16 1:27:11 by hurstdog]: oddly, restarting apache fixed it. (knock on wood). You may go back to reading k5 sans error messages now :)

First, the thank yous

In the first days of the disaster, Jason Camp of VHosting worked tirelessly trying to get bubba to come back from the dead. To no avail, unfortunately, but both for that, and for all the great support he's given us for the past year-and-a-few-months, I am eternally in his debt.

Voxel.net stepped in with an offer of new hardware and redundant cluster hosting, so that future single-machine failures won't have this kind of effect. It was a scheme we've been wanting to pursue for a while, and they have a lot of experience with highly-available server clusters, so we jumped on that. They have worked tirelessly, and provided probably the best support I've ever seen anywhere in getting this new setup going. And it has not been an easy road.

Specifically, I have to thank Raj, Voxel's President, who went so far as to call me from Singapore, where he's on vacation, to let me know what the status of the project was. Andres, who wrestled with LVS last night until 6AM, finally getting it to work properly, and Jim, who put in similarly long hours and great effort overcoming what has surely been the most problem-ridden installation in history.

Anybody can do something and have it go perfectly right off. What really shows you what people are made of is how they act when everything goes wrong, and these guys are a class act.

Finally, I have to thank our still-anonymous no longer anonymous (!) friend, el_guapo, at Compaq, who also went to great lengths to rescue our data from bubba's arrays. I'd love to name in if he were to give me permission to do so. But until then, he remains our anonymous friend.

So what's going on now?

Right now, the site is running on a total of three machines. One lightweight box in front is running LVS, which is a Linux load-balancing system. It's splitting requests off to two dual P700s, one of which is running Apache and Scoop, and the other of which is running Apache, Scoop, and MySQL. Both are using the one database, and LVS is configured to favor the scoop-only machine for web connections, to compensate.

All this is because the database machine (formerly known as "hex" for those of you who keep up) is having kernel trouble. Very soon, it will come back online, and there will be a brief outage while we move the database back onto it's own box. Really, it'll be minutes. I promise. ;-)

The Future

Once the four-machine cluster is running smoothly, we will be swinging into action to build it up and ensure full redundancy. Promicro Systems, who built the two new Scoop machines we're running on now, is going to become a hardware sponsor, and hook us up with some more new machines, to fill in wherever the current system isn't fault-tolerant. We hope to end up with:

  • Two quad Xeon database servers, one live, one mirroring the live one, as a hot spare
  • Four Scoop servers, dual P3 class
  • Two LVS frontends, one hot-spare which will take over if something happens to the live one
This should bring us to a point where the cluster can expand as needed, and won't be so fragile with respect to hardware failure.

And yes, from now on, we're going to keep backups.


Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure


Related Links
o Scoop
o Operation Enduring Downtime
o VHosting
o Voxel.net
o Promicro Systems
o Also by rusty

Display: Sort:
Well. That Sucked. | 105 comments (105 topical, editorial, 0 hidden)
first post (4.73 / 19) (#1)
by fluffy grue on Sat Dec 15, 2001 at 04:17:00 PM EST

Hey, if you're allowed to break story posting policy... :D
"Is not a quine" is not a quine.
I have a master's degree in science!

[ Hug Your Trikuare ]

By the way (5.00 / 7) (#2)
by rusty on Sat Dec 15, 2001 at 04:19:26 PM EST

I know that it's bad form to have posted this to the Front Page manually, and I haven't turned into a power-mad demigod during the last month, I just thought it was pretty important to have something to greet returning readers with.

I promise it won't happen again, barring similar extraordinary circumstances.

Oh, and note #2: This is obviously a whole new system, and there may be problems we haven't forseen. Please forgive any strangeness you encounter, and perhaps post a note here if something's unusually broken.

Not the real rusty

Yay & Time (5.00 / 2) (#7)
by duxup on Sat Dec 15, 2001 at 04:34:20 PM EST

1. You're forgiven, thanks for the hard work during the downtime. I hope operation uptime will be as much or more a success than the previous operation.

2. It would seem that when I posted my first diary the time stamp for the diary is off by about two hours (my display is set for CST).

[ Parent ]
yep (5.00 / 2) (#9)
by hurstdog on Sat Dec 15, 2001 at 04:39:17 PM EST

The servers are off by 2 hours, and not synced. Since we were focused on getting them up, we didn't notice :) Anyway, we're going to install ntp and get them synced within the cluster. That will come within the week.

[ Parent ]
Time problems (5.00 / 2) (#10)
by rusty on Sat Dec 15, 2001 at 04:39:34 PM EST

Yes, there are time issues. As I mentioned, "unexpected problems" might crop up, and here they are.

The two running boxes are both two hours behind EST (where they should be), and are out of sync by 8 minutes or something. This out of syncness is causing some weird behavior with comment counts and marking "new" comments. We'll have it fixed up in a jiff. Or two jiffs, perhaps.

Not the real rusty
[ Parent ]

Right (4.66 / 3) (#3)
by DJBongHit on Sat Dec 15, 2001 at 04:20:36 PM EST

Fat lot of good it does me now. I have a very eventful 2 weeks, and I don't even have my k5 diary to let people who don't care know what's going on!


GNU GPL: Free as in herpes.

That's what your ~. diary is for (none / 0) (#75)
by mikael_j on Sun Dec 16, 2001 at 04:49:01 PM EST

We give a bad name to the internet in general. - Rusty
[ Parent ]
To hell with Rusty... (5.00 / 2) (#4)
by issue9mm on Sat Dec 15, 2001 at 04:24:04 PM EST

who was supposed to notify me of the 'golive' time so that I could get first post.

It's okay though, I guess, seeing as how he's going to get voxel.net to host his adoring cousin for free as well. ;)


Nyah (none / 0) (#26)
by fluffy grue on Sat Dec 15, 2001 at 08:19:38 PM EST

"Is not a quine" is not a quine.
I have a master's degree in science!

[ Hug Your Trikuare ]
[ Parent ]

Mya! (none / 0) (#27)
by enani on Sat Dec 15, 2001 at 08:20:50 PM EST

--=* Moshi-moshi! *=--
[ Parent ]
Congratulations (2.00 / 1) (#5)
by abo on Sat Dec 15, 2001 at 04:29:06 PM EST

Congratulations, you got it up and running again before Google ha a chance to fail to re-index the site! Is that good news about Kuro5hin or bad news about Google?
-- Köp BRUX!
Yay!!! (none / 0) (#6)
by Mad Goose on Sat Dec 15, 2001 at 04:29:20 PM EST

Finally back. I was going into K5 withdrawl...you know the symptoms...bloodshot eyes, runny nose, headaches, removing everything except your tighty-whiteys and shaking in the corner... ;)
Seriously, though, big thanks to all of you who brought K5 back to life. Now I can post my entirely silly diaries again!

How do you know this post isn't the result of a drunken bet?

Discworld "Map":
"There are no maps. You can't map a sense of humor."
-Terry Pratchett
with*drawl*? (5.00 / 2) (#47)
by j3z_ on Sun Dec 16, 2001 at 12:59:00 AM EST

ee-yas tha-yt suhm-thin ye-ou ge-yt veh-ry arrf-en? ye-ou ahh-ought'y gee-chyou say-lf too ay daaah-ctor!

[ Parent ]
No more vacations for you mister! (5.00 / 4) (#8)
by wiredog on Sat Dec 15, 2001 at 04:36:27 PM EST

Man, every time you got off somewhere unreachable, Bad Things Happen.

The idea of a global village is wrong, it's more like a gazillion pub bars.

No kidding! (none / 0) (#11)
by rusty on Sat Dec 15, 2001 at 04:41:03 PM EST

Now I feel like I can never go on vacation again. Or, like, I can only travel where they have good broadband. :-/

Not the real rusty
[ Parent ]
What's up w/that anyway? (none / 0) (#12)
by wiredog on Sat Dec 15, 2001 at 04:45:20 PM EST

Is there some deep psychic connection between you and whatever server is running so that it knows when you are away? Was Scoop abused/ignored in its infancy, so that it now tries extra hard to get Daddy's attention? Does poor little Scoop have separation anxiety?

Is it the Terrible Twos? If so, what's Scoop gonna be like when he's a teenager?

The idea of a global village is wrong, it's more like a gazillion pub bars.
[ Parent ]

Ockham's razor (5.00 / 1) (#17)
by Defect on Sat Dec 15, 2001 at 07:28:43 PM EST

Two people who regularly access k5's box, rusty and inoshiro. When rusty leaves, k5 dies.

The obvious, and only reasonable, answer is that inoshiro is brainwashed by the canadian government to take down k5 when there's no other supervision due to the fact that k5 is anticanadian and bad. Ask legolas about k5's policy on canadians.
defect - jso - joseth || a link
[ Parent ]
Do you think... (4.50 / 2) (#18)
by wiredog on Sat Dec 15, 2001 at 07:32:38 PM EST

Inoshiro is the one who poisoned Rusty with Girl Scout cookies a while back? Or is Driph the likelier suspect?

The idea of a global village is wrong, it's more like a gazillion pub bars.
[ Parent ]

The secret? (5.00 / 1) (#33)
by rusty on Sat Dec 15, 2001 at 09:30:39 PM EST

I keep K5 running by pure force of Will. It's obvious. Even I doubted it, at first, but there can be no other explanation. Just look at the facts...

Not the real rusty
[ Parent ]
So (none / 0) (#13)
by wiredog on Sat Dec 15, 2001 at 07:14:44 PM EST

How many of us have submitted the news to /.?

The idea of a global village is wrong, it's more like a gazillion pub bars.

RE: So (none / 0) (#32)
by hyyx on Sat Dec 15, 2001 at 09:09:54 PM EST

I submitted, but after I did I regretted it. It is not news worthy of an entire discussion and it will probably get rejected (why would /. want to promote K5 and possibly lose people). I'm sticking around here while it's still up ... :)

[ Parent ]
And that brings us back to do... (none / 0) (#95)
by zantispam on Mon Dec 17, 2001 at 10:54:09 AM EST

why would /. want to promote K5 and possibly lose people

Because they like us. Besides, that's where most of us came from to begin with.


Free Duxup!
[ Parent ]
Bug. (none / 0) (#14)
by Anonymous 6522 on Sat Dec 15, 2001 at 07:17:59 PM EST

How come this story isn't showing up under Site News in the Section Stories box?

caching (5.00 / 1) (#15)
by hurstdog on Sat Dec 15, 2001 at 07:19:39 PM EST

Its too much work on the db to compute that every page load. So its only done every 1000 hits or so. Thus, give it a bit, and it will show up.

[ Parent ]
Because it's on the front page? (n/t) (5.00 / 1) (#16)
by wiredog on Sat Dec 15, 2001 at 07:20:27 PM EST

The idea of a global village is wrong, it's more like a gazillion pub bars.
[ Parent ]

Welcome back (none / 0) (#19)
by derek_m on Sat Dec 15, 2001 at 07:32:55 PM EST

I hadnt realised how many times I tended to visit k5 on auto-pilot until the recent problems - the number of times I saw the temporary page "accidentally" was ridiculous.

So - welcome back, thanks to all for the hard work, and lets hope you dont have to deal with any more major server/hosting related hassles for a long time.

K5 is BACK (none / 0) (#20)
by scorpion on Sat Dec 15, 2001 at 07:44:08 PM EST

Glad to see you back. I can now settle down and relax with my daily read.

Oh well, I missed K5 (none / 0) (#21)
by mami on Sat Dec 15, 2001 at 07:44:27 PM EST

What a bloody shame.

Kudos to the guys who brought it back.

Very snappy! (none / 0) (#22)
by theantix on Sat Dec 15, 2001 at 07:57:47 PM EST

Now, it could be just because there aren't as many people as usual because they don't know that you're up... but the current K5 setup seems extraordinarily snappy. Very quick, almost scary quick! =) Good job, glad to see everyone starting to come on back.

You sir, are worse than Hitler!
Well... (none / 0) (#31)
by rusty on Sat Dec 15, 2001 at 09:08:57 PM EST

It's Saturday night. This is the slowest time of the week anyway, and word still probably isn't out. I don't expect the crrent system to stand up to what we expect normal load to be.

Luckily, this is very temporary, and it just improves from here. I'd say within two weeks we'll be bulletproof. Or at least Slashdotproof, which is more important. :-)

Not the real rusty
[ Parent ]

Gott in Himmel! (none / 0) (#23)
by DesiredUsername on Sat Dec 15, 2001 at 08:02:30 PM EST

Dot vas no goot!

I got work done for three weeks! And I'm just barely getting paid, so that's no damn good! Where were you, K5, in my time of need??

Play 囲碁
Story queue... (4.00 / 1) (#24)
by chipuni on Sat Dec 15, 2001 at 08:04:52 PM EST

You know... I feel sorry for the twelve people who have been waiting anxiously for a month to find out whether their stories would be accepted!

Seriously, it's wonderful to see Kuro5hin back!
Perfection is not reached when nothing more can be added, but only when nothing more can be taken away.
Wisdom for short attention spans.

Doing something about it (4.50 / 2) (#25)
by Erbo on Sat Dec 15, 2001 at 08:10:18 PM EST

That's why I went in and immediately voted on everything in the queue, to try and flush out the backlog. If you haven't done so, go do it.

Electric Minds - virtual community since 1996. http://www.electricminds.org
[ Parent ]

Could you take it down again for a few days? (2.00 / 1) (#28)
by roystgnr on Sat Dec 15, 2001 at 08:40:31 PM EST

Don't get me wrong, I've had some serious withdrawal symptoms (God help me, I read adequacy...) and it's wonderful to have kuro5hin back, but at the same time I could actually feel my GPA rising when I had all sorts of free time to spend on school. Could we take the site down for just another day or two until I've finished my last final? Thanks.

i second that (none / 0) (#30)
by Delirium on Sat Dec 15, 2001 at 08:55:03 PM EST

Kuro5hin coming up two days before my first final, when I still have 3 essays to write, is pretty crappy timing. =P

But thanks anyway rusty!

[ Parent ]

What's wrong with Adequacy? (1.25 / 4) (#36)
by Patrick Bateman 10005 on Sat Dec 15, 2001 at 09:42:51 PM EST

Adequacy is one of the most insightful and thought-provoking discussion sites on the entire Internet.

[ Parent ]
Nothing's wrong with Adequacy (none / 0) (#37)
by roystgnr on Sat Dec 15, 2001 at 10:14:10 PM EST

It's a beautiful site; it turns a malicious pasttime into a true artform. There have been comp.lang.c trolls, Slashdot trolls, etc... but Adequacy is trolling the entire internet. And to judge by the comments, successfully!

[ Parent ]
For your information: (5.00 / 1) (#42)
by tyronefine on Sat Dec 15, 2001 at 11:31:49 PM EST

On adequacy, the "biters" are the editors responding to their own stories. Hope I didn't spoil your enjoyment.

it turns a malicious pasttime into a true artform

If by "turning it into a true artform" you mean "sucking all the life out of it by reducing it to tiresome formulaicism", you are correct.


[ Parent ]

Nope. (none / 0) (#46)
by Estanislao Martínez on Sun Dec 16, 2001 at 12:26:49 AM EST

On adequacy, the "biters" are the editors responding to their own stories.

Nope. And we have the server logs to prove it.

[ Parent ]

Where did you find those guys? (4.50 / 2) (#57)
by Nick Ives on Sun Dec 16, 2001 at 04:01:28 AM EST

I'm curious, when I first saw the number of comments on that story I thought for sure that you were undergoing a DoS attack or something, then I loaded it up and saw that there was indeed 2000 (now its at about 3.5k) or so unique comments. The only thing that surprised me more is that they were all saying the same thing. Where did you manage to find a group of people who are so dimwitted as to think saying the same things over and over again ad infinitum makes a solid argument? At first I thought it must have gotten posted to /., but I couldnt find any mention of it there. I've seen links to adequacy articles posted on usenet in the past and I figure you guys send out links to your stories to relevant sites/blogs. Between that and your referer logs you must have some idea where they all came from.

Inquiring minds want to know...

Unless they all just came from disparite corners of the 'net and you happened to luck out on the stupidity quotient.

[ Parent ]

They came on their own. (none / 0) (#59)
by Estanislao Martínez on Sun Dec 16, 2001 at 04:30:17 AM EST

The story just exploded on its own. People are *still* posting the link all over the web-- and people replying "Oh, that's old already, under what rock did you hide all this month?"

[ Parent ]

Adequacy.org: (none / 0) (#44)
by Anonymous 6522 on Sat Dec 15, 2001 at 11:45:39 PM EST

They troll the entire internet like spammers sell penis enlargers to the entire internet.

[ Parent ]
See you in court. (none / 0) (#55)
by Estanislao Martínez on Sun Dec 16, 2001 at 02:50:10 AM EST

They troll the entire internet like spammers sell penis enlargers to the entire internet.

Google Groups search for "enlarge your penis": 34,800 results.

Google Groups search for "Adequacy.org": 609 results.

Expect papers from our lawyers soon. That IPO money isn't just for hardware, y'know.

[ Parent ]

Don't make me stop this car, you two! (5.00 / 1) (#56)
by rusty on Sun Dec 16, 2001 at 03:04:08 AM EST

Not the real rusty
[ Parent ]
You proved my point. (5.00 / 1) (#58)
by Anonymous 6522 on Sun Dec 16, 2001 at 04:03:02 AM EST


Estanislao Martínez: Proving your points for you since 1981.

[ Parent ]

The best thing on adequacy (none / 0) (#90)
by lucius on Mon Dec 17, 2001 at 04:39:01 AM EST

is definitely tkatchev. He alone is worth reading the site for.

[ Parent ]
It's boring (5.00 / 3) (#40)
by Delirium on Sat Dec 15, 2001 at 11:08:14 PM EST

The stories are all in the same style. It was funny when qpt posted some stories in that style to k5 a year ago. It's not funny anymore after hundreds of nearly-identical stories are posted over and over again. Just because one style of satire worked once, doesn't mean all you have to do is change the topic and it'll keep working ad nauseum...

[ Parent ]
I hate (none / 0) (#50)
by toga98 on Sun Dec 16, 2001 at 01:53:32 AM EST

Yeah, I agree. What's wrong with spreading misconceptions, stereotypes, and a bit of unhealthy ignorance. Seriously, if you're doing research on race relations, adequacy is a great place to start.

The Holocaust is primarily a historical hoax and/or the greatest holocaust was not against Jews but perpetuated on Christians by Jews
-David Duke, Ku Klux Klan

[ Parent ]
Bah. (none / 0) (#39)
by regeya on Sat Dec 15, 2001 at 10:59:06 PM EST

During usual times, I spend about 15min. at work reading kuro5hin, mostly spent speed-reading new stories.

During the last three weeks, I've been (oh, no, am I about to be the next Lee Maletesta? :-P Feds, I live on 1313 Mockingbird Lane, in Dullsville, MD...) doing stuff like looking for stuff via LimeWire. Enterprise episodes, anyone?

So now, with kuro5hin back up, I can get some more work done... ;-)

[ yokelpunk | kuro5hin diary ]
[ Parent ]

Get Enterprise here! (none / 0) (#82)
by twodot72 on Mon Dec 17, 2001 at 01:49:40 AM EST

D00D, you can dl all Enterprise eps from me, join in the fun at ftp.warez.fbi.gov, THE premier WAREZ site on the Internet!!!

[ Parent ]
Heh (none / 0) (#51)
by rusty on Sun Dec 16, 2001 at 02:07:08 AM EST

Hasn't anyone noticed yet that mentioning Adequacy inevitably starts the pile-on?

For good or ill, they sure sem to have become the web's most controversial site. To the Adequacy detractors, I'd just have to say don't take satire so seriously, and especially not when it's satirizing you. :-)

Not the real rusty
[ Parent ]

*gasp* IT IS ALIVE! (none / 0) (#29)
by WWWWolf on Sat Dec 15, 2001 at 08:49:05 PM EST

Don't believe these "I got my work done, please shut it down again!" folks! While K5 was down, I could waste my time very well without K5, too!

Anyway, it's nice to see K5 back...

...at least until some readers of the Other Site notice it and it this message of joy gets posted there... =)

-- Weyfour WWWWolf, a lupine technomancer from the cold north...

RDF feed broken (4.50 / 2) (#34)
by ikarus on Sat Dec 15, 2001 at 09:31:52 PM EST

probably just got lost somewhere in the setup, but the RDF feed doesn't seem to be updated/updating.

System Power (4.50 / 4) (#35)
by tzanger on Sat Dec 15, 2001 at 09:38:23 PM EST

Out of curiousity I'd love to know if the power of the systems in place is really necessary... Dual P3/700s for serving up pages? A dual P3/700 for MySQL, which 99% of the time is running simple selects and inserts (IIRC)?

I've never set this kind of stuff up before but from my own office networks I've noticed that processing power is hardly necessary except when, heavy processing is required. Hell my main fileserver (over 100 active clients) was a P90 with a killer I/O system until the new SCSI RAID controller required a more up-to-date PCI controller. The database server needed some more balls because the queries are quite complex. The mailserver pushes about 30G/mo but just needs lots of RAM for the spam check; processor power is vastly underutilized even on the P2/233.

This comment isn't meant to be offensive, but rather out of curiosity. What does the processor load look like and how does it correlate to pageviews? I heartily believe in clusters and redundancy; I'd like to know if cheaper (lighter) hardware would work.

Well. (none / 0) (#63)
by mindstrm on Sun Dec 16, 2001 at 10:40:18 AM EST

I obviously can't speak for K5... but in my own experience, your assumption is correct.

The processor speeds we see today are generally not necessary for many of the tasks we do. THe same goes for home computers.

But why a Dual 850 instead of, say, a dual 500? BEcause.. that's all you can buy. You don't save any money by buying a pIII 500.

[ Parent ]
And of course there's upgrade cycle (none / 0) (#91)
by RadiantMatrix on Mon Dec 17, 2001 at 04:55:53 AM EST

You don't save any money by buying a pIII 500.
There's also the issue of controlling your upgrade cycle. If a P3/850 is $30 more than a P3/500, you're better off with the faster machine because (theoretically) your system can expand more before you need to upgrade.

Since upgrades are usually when the most downtime occurs, upgrading less means less downtime too.

No amount of genius can overcome a preoccupation with detail.

[ Parent ]

The Power Issue (5.00 / 2) (#65)
by rusty on Sun Dec 16, 2001 at 11:09:55 AM EST

This has come up before -- genrally every time we upgrade hardware, someone asks why we need it. :-)

Basically, Scoop is really, really dynamic. If all this was static html pages, we could run it off a single PPro. But to do all the nifty stuff that Scoop does, it has to hit the database really hard.

Processor power is not really the key factor, you're right. Memory is, though, and machines that support a lot of memory tend also to be SMP. Besides which, the more processors you have, the more httpd and mysql threads you can run, and the better off you are. That is, more slower processors are better, for this application, than fewer faster processors.

Right now, one of the scoop boxes has died, so everything's running on a single dual P3 750 with a gig of RAM. On a Sunday morning, the load is around 8-9. We really do need this stuff. :-)

Not the real rusty
[ Parent ]

Cool, but... (5.00 / 1) (#67)
by tzanger on Sun Dec 16, 2001 at 12:35:04 PM EST

Basically, Scoop is really, really dynamic. If all this was static html pages, we could run it off a single PPro. But to do all the nifty stuff that Scoop does, it has to hit the database really hard.

Right; that is database. That's what I thought...

So I ask again, perhaps clearer: Why not use the dual fast-as-hell, great I/O and memory server for the database? Maybe a couple single-proc Celerons or cheap-AMDs just for DB reads (and a move to Postgres, ooh a flame war!) and then a battery of cheap-as-you-can-buy servers for the apache frontends? I've found backplane systems that fit 4 systems into a 4U case... Hell you could fit the image/ad server off of one of the db servers since it'd be all static serving (ad image is selected by scoop and the image is statically selected from the image server)...

Is there any caching done in Scoop to try and keep the 10-zillion-exactly-the-same-DB-hits to a minimum? That is one thing I think /. really got right: frontpage updates are part-static, part-based-on-user, and even has static story views, although I've been bugging them to make the static views "nested" instead of "threaded" so I can read the whole thing when the DB is going crazy. :-)

Of course, there are a dozen ways to do it, each with their own plusses and minuses. I know K5 is ad-funded+sub-funded+any-donations-welcome so spending a pile on hardware is out of the question, which is why I was thinking of the more-cheaper-system approach. Anyway like I said, I was just curious and thought I'd ask. I'm really glad K5 is back. :-)

aside: is it possible to set up a stat page where the readers can see the hits/load/mem/cluster status broken down by time/day? It'd be a great reference just to see how a realworld system works.

[ Parent ]
Oh, I see (5.00 / 2) (#69)
by rusty on Sun Dec 16, 2001 at 01:06:22 PM EST

Why not use the dual fast-as-hell, great I/O and memory server for the database?

I see what you were getting at. In fact, the dual P3 700s are the slow machines in all this. Only the LVS is lighter-weight. Right now, we have three boxes that are all about the same-- that dual P3 class. Soon, those will all be running scoop, and the database will be on a quad Xeon. Your strategy is right, your standards are just too low. :-)

If we were buying servers, of course, we'd have none of this nice stuff. Luckily, companies like Promicro (plug! plug!) see the adavantage in giving us hardware in exchange for getting their name in front of all of you.

I mean, what we could make do with is one thing, but would you turn down better stuff if offered?

Is there any caching done in Scoop to try and keep the 10-zillion-exactly-the-same-DB-hits to a minimum?

Yes, quite a lot actually. If you pay attention, and can pinpoint when we restart apache, you'd notice that the first few pageloads are pretty slow, but it ramps up rapidly. Scoop caches everything it possibly can. The tradeoff for this is it hogs memory. The more memory, the better the performance. It's almost linear.

About stats: I will try. It would be interesting, wouldn't it?

Not the real rusty
[ Parent ]

Stats Page (none / 0) (#93)
by fuzzcat on Mon Dec 17, 2001 at 09:06:27 AM EST

I would be really interested in this, honestly. I'm just setting up my own very first webserver, and it would be interesting to me to compare the loads.

So, yeah...If you could get this working, I know that I would be quite interested in it.

[ Parent ]

Welcome back (3.00 / 2) (#38)
by Begbie on Sat Dec 15, 2001 at 10:53:35 PM EST

Ah, now I can read so real discussion, instead of the trite that passes as discussion on certain other other sites.

Glad that you're back.

Voxel.net GIF (none / 0) (#41)
by J'raxis on Sat Dec 15, 2001 at 11:26:15 PM EST

Yackhh is that thing bright. Something about the orange surrounded by a gray, white, black, and steel blue color theme. It really just stands out too much.

— The Raxis

[ J’raxis·Com | Liberty in your lifetime ]

yeah (4.00 / 1) (#49)
by rusty on Sun Dec 16, 2001 at 01:37:36 AM EST

We don't like it much either. We'll get something nicer looking in there soon.

Not the real rusty
[ Parent ]
Well... (none / 0) (#60)
by Anonymous 6522 on Sun Dec 16, 2001 at 04:35:10 AM EST

It's better than that brown one you had for awile. I say you stick with the first one you had up. You know, the one that said something along the lines of, "Voxel dot net: They kick ass."

[ Parent ]
Well... (none / 0) (#64)
by Legolas on Sun Dec 16, 2001 at 10:52:22 AM EST

If that sort of stuff really bothers you (like the OSDN crap bothered me), the logical solution is to do something about it, and get a filtering proxy.

My particular favorite is Proxomitron (for Win32), since it is very versitile and comes with lots of neat built in rules. With a half an hour work, you can transform Kuro5hin into something like this.

I may publish a guide on creating new rules in proxy filters, as to make your favorite websites look better, sometime in the near future.


[ Parent ]
The ads don't bother me. (none / 0) (#71)
by Anonymous 6522 on Sun Dec 16, 2001 at 01:59:20 PM EST

I just thought that first text one was a whole hell of a lot cooler than all the others, and the brown one was just plain ugly.

The only ads that piss me off are pop(ups/unders), and turning off javascript takes care of that. Those news "shoshkels" (or whatever) have the potential to get really annoying, but I don't use IE, so they shouldn't work. Even if they do manage to get them to work on whatever browser I'm using, I'm willing to mangle it's functionality to get rid of them.

[ Parent ]

Pity you don't run Linux... (none / 0) (#83)
by ramses0 on Mon Dec 17, 2001 at 02:14:44 AM EST

Galeon kicks ass as a web-browser, it's got two amazingly cool settings- "right-click, block images from this server", and "settings -> Allow Popups?". A lot of people also like the 'tabbed browsing' feature, but it doesn't quite work with my work-style.

I used to give my friends heck who ran linux, saying "oh, if it's so great for working with the internet, then why do all of the web browsers suck?" Happily enough, some awesome people have done something about it, leaving us with uber-crossplatform mozilla, k-rad K-meleon (browser, file manager, etc...), galeon (it just works), opera (i'd rather pay for beer than not have it at all), and of course our old text-mode standbys, lynx and wget. ;^)

It's an awesome time to be running with the Linux crowd right now. :^)

[ rate all comments , for great ju
Parent ]

Galeon rules (none / 0) (#86)
by rusty on Mon Dec 17, 2001 at 02:30:55 AM EST

I have switched to it full-time. I love the tabs, personally. I can leave popups enabled, and just x-off the ones I didn't ask for, without losing focus, and without losing the ability to open those crappy javascript links when it becomes necessary. And I've already shut off images from many an adserver. It's so easy. See an ad? Just select "block from this server" in the right-click menu. Mmmm.

There seems to be a gecko bug that affects nested comment view here though. Sub-comments are shifting out farther than they ought to.:-/

Not the real rusty
[ Parent ]

GIF (none / 0) (#105)
by J'raxis on Fri Dec 21, 2001 at 07:13:11 PM EST

The one you have now is excellent. Thanks for making it less obtrusive.

— The Raxis

[ J’raxis·Com | Liberty in your lifetime ]
[ Parent ]

Unfortunately... (4.00 / 1) (#66)
by theboz on Sun Dec 16, 2001 at 12:05:05 PM EST

If you go to their site it looks like their logo does use orange for the text. I think the k5 crew should just use a smaller version of their logo, like this for the time being, but voxel could definitely use some help with their graphics. I do have to say that their site is good for presenting information rather than a bunch of flash and graphics though.

[ Parent ]

I did (none / 0) (#77)
by J'raxis on Sun Dec 16, 2001 at 05:53:29 PM EST

I did go to their site. But I also noticed they use a blue theme around their site similar to the K5 colors (for example), and a neutral black or gray could also be an option.

Is that specific orange logo their registered trademark? (I don’t see a TM or an ® on it, nor on the one on their site.) If it were, then I the colors would play a major part in the logo, but it does not seem to be.

— The ®axis

[ J’raxis·Com | Liberty in your lifetime ]
[ Parent ]

I hope it's not a trademark logo (4.00 / 1) (#81)
by theboz on Sun Dec 16, 2001 at 11:12:33 PM EST

Their logo reminds me a lot of the AT&T deathstar logo too. I think what happened is that someone that works there decided they needed some sort of logo so they opened up photoshop/gimp or whatever and just made something very simple. The orange color looks ok on black, but not on white. When choosing colors for your logo people should pick something that looks good especially on black or white, because while their webpage might have a black background there, their business cards and stationary will have a white back ground, and it doesn't look as professional to keep the logo in a black box.

Of course, I have never done any professional art or graphics jobs, so what do I know? :-)

[ Parent ]

Welcome Back! (none / 0) (#43)
by martinchmiel on Sat Dec 15, 2001 at 11:42:16 PM EST

Good job K5 team! I look forward to K5 once again..
martin.. waterloo.ontario.canada
Hey Rusty (5.00 / 1) (#45)
by Vladinator on Sat Dec 15, 2001 at 11:51:33 PM EST

It's good to see you guys back up! We were all pulling for you over at GiZ. Congrats on the hardware coming together!
LRSE Hosting
Seconded! (none / 0) (#61)
by Erbo on Sun Dec 16, 2001 at 04:47:16 AM EST

I agree, it is good to see you guys back up. I know first-hand how frustrating downtime can be...and even now, we're having problems with one of our DNS names...

Here's to nice long uptimes in the future!

Eric J. Bowersox
Technical Director, Electric Minds Community (http://www.minds.com)
Electric Minds - virtual community since 1996. http://www.electricminds.org
[ Parent ]

Did anyone just notice... (none / 0) (#48)
by hyyx on Sun Dec 16, 2001 at 01:08:58 AM EST

[sometime around 12pm, just waking up]...

Hey. Great. K5 is back, I am REALLY happy.


This is really great. Getting back into the K5 mode.


... damn where'd K5 go ? ... Secret Service ...

heh (5.00 / 3) (#52)
by el_guapo on Sun Dec 16, 2001 at 02:18:41 AM EST

well, rusty'd been keeping it a secret, and i've always thought everybody knew. must've generated a few interesting/humorous #k5 discussions, since i thought everyone knew. anyways, enough DO know that to continue keeping a "secret" seems pointless. my 1 request is that i don't get inundated with free hardware requests. oh yea - and go to www.solutionscenters.tv to see bubba, he's waaaaaay off to the right where that CRT is. (elguapo officially releases rusty from his confidentialty aggreement :-) (since i had already blew it, unbeknownst)
mas cerveza, por favor mirrors, manifestos, etc.
That webcam of yours... (3.00 / 1) (#73)
by banstyle on Sun Dec 16, 2001 at 04:29:14 PM EST

You know, love the web cam. Right now, Sunday at 3:19 your time, I'm watching an AB-Doer commercial on your TV, but I do see bubba! (c:

PS: 1-800-255-2424 to order!

"Everything done in weakness fails. Moral: do nothing." -Nietzsche
[ Parent ]
What I did on my K5 vacation ... (none / 0) (#53)
by joegee on Sun Dec 16, 2001 at 02:33:02 AM EST

I went to back to an old haunt that grew old again, in a hurry.

I renewed contacts with some geek acquaintances at one of my favorite hardware sites.

I spent some time back at one of my old sources of chit chat. Unfortunately it's still under the draconion control of England's thought police, the BBC.

I burned all my MP3s to CDs. I went through my Windoze registry and familiarized myself with a few more keys. I organized my burned CDs. I brushed up on my Red Alert II game. I cleaned my keyboard.

Rusty, Inoshiro, et al -- I am very happy K5 is back. Thank you. :)

<sig>I always learn something on K5, sometimes in spite of myself.</sig>
H2G2.com without the BBC controls (4.00 / 1) (#68)
by pin0cchio on Sun Dec 16, 2001 at 12:49:37 PM EST

I spent some time back at one of my old sources of chit chat [h2g2.com]. Unfortunately it's still under the draconion control of England's thought police, the BBC.

If you want your H2G2 collaborative encyclopedia fix, but you don't want the Beeb interfering, go to Wikipedia or Everything 2. I don't post on either site, but I have let Damian Yerrick use some of my stuff on E2.

[ Parent ]
Ahem, *English*? (none / 0) (#94)
by gordonjcp on Mon Dec 17, 2001 at 09:44:27 AM EST

I spent some time back at one of my old sources of chit chat. Unfortunately it's still under the draconion control of England's thought police, the BBC.
What does the first 'B' in 'BBC' stand for?
'Britain' is not 'England'. England is only one country, that makes up part of Britain.
I know most Americans have only recently learned that there are other countries in the world, but please try and get this right...

Give a man a fish, and he'll eat for a day. Teach a man to fish, and he'll bore you rigid with fishing stories for the rest of your life.

[ Parent ]
You Brit's are so cute ... (none / 0) (#101)
by joegee on Mon Dec 17, 2001 at 06:08:57 PM EST

... when you're being sanctimonious.:) You know, in spite of Tony Blair I can still come in here and find reason to believe you're every single bit as obnoxious as you accuse us of being. :)

Where did the BBC originate? Scotland? Wales? The Isle of Wight? Silly me. Here I thought London was just in another state where everyone spoke funny.

<sig>I always learn something on K5, sometimes in spite of myself.</sig>
[ Parent ]
And here I thought Britain was wrong... (none / 0) (#103)
by Delirium on Tue Dec 18, 2001 at 03:26:08 AM EST

...because Britain is only one part of the United Kingdom of Great Britain and Northern Ireland.

[ Parent ]
draconiAn ... Yes, yes, I saw it too late. (NT) (none / 0) (#54)
by joegee on Sun Dec 16, 2001 at 02:34:47 AM EST

<sig>I always learn something on K5, sometimes in spite of myself.</sig>
What about that OTHER downtime story? (4.66 / 12) (#62)
by sakusha on Sun Dec 16, 2001 at 05:45:34 AM EST

OK, you're back. Now maybe you'll have time to write up that story that was on the temp page, about the Secret Service or FBI or whoever, interrogating K5 users.

Just one question (3.00 / 4) (#70)
by CmdrTroll on Sun Dec 16, 2001 at 01:20:50 PM EST

Given that you pretty much rebuilt the site from scratch, I've got to ask: why didn't you take this opportunity to move up to FreeBSD?

It took the loss of an important filesystem to the hack they call "ext2" before I learned my lesson, but I'm much happier now. The ports tree, soft updates, better scalability, and releases that aren't <a href="http://www.uwsg.iu.edu/hypermail/linux/kernel/9809.2/0111.html">constantly</a> <a href="http://groups.google.com/groups?q=linux+filesystem+corruption+2.4.15&hl=en&rnum=3&selm=linux.k
ernel.20011123203452.A3141%40flint.arm.linux.org.uk">broken</a> clinched the deal for me.


Threading (none / 0) (#76)
by hurstdog on Sun Dec 16, 2001 at 04:58:12 PM EST

I've been pushing for FreeBSD for a while now :-) But I think we didn't this time because of threading. Supposedly MySQL doesn't like the threading on the BSD's. So, until the site runs PostgreSQL ( which is after I port it and we find a good reason to move to it, which will be a while ;) or the BSD's change the way they do threading, we're on linux.

At least thats how I understand it.

[ Parent ]
BSD Threading (none / 0) (#84)
by Miniluv on Mon Dec 17, 2001 at 02:17:51 AM EST

OpenBSD has fixed this issue, resulting in something in the area of a 60% (x?) performance increase. Story was on Root Prompt a few days ago. I'd link it, but I'm far too lazy.

Applied Solipsism worked for me.
[ Parent ]

OpenBSD (4.00 / 1) (#85)
by locke baron on Mon Dec 17, 2001 at 02:26:20 AM EST

OpenBSD doesn't do SMP, though (and neither does NetBSD, at least on x86, as near as I can tell), and all the machines in the cluster are SMP. Once they pull their socks up on this, though, I can't say I'll be able to beef too loudly (and may just have to install OpenBSD on my firewall, which is a dual-proc P133, currently running Linux).
Micro$oft uses Quake clannies to wage war on Iraq! - explodingheadboy
[ Parent ]
Good point (none / 0) (#87)
by Miniluv on Mon Dec 17, 2001 at 02:38:18 AM EST

But then again, no offense to the FreeBSD guys, but FreeBSD SMP is as close to useless as functional could ever get.

However, the unspoken, and unhinted at even, point of my post was that if OpenBSD fixed their thread implementation to work better with MySQL perhaps the FreeBSD folks, with proper prodding, could adapt said functionality and make the world a kinder, gentler place for open source relational databases.

Applied Solipsism worked for me.
[ Parent ]

Good point... (none / 0) (#88)
by locke baron on Mon Dec 17, 2001 at 03:10:16 AM EST

I'm not entirely sure what the FreeBSD guys are dinking with just now... Maybe they have fixed this stuff in 5.0 (which, from what I'm told, actually does SMP better than Linux, and better than Solaris on x86 rigs)... Of course, 5.0 is still beta (alpha?) so this is all moot.
Micro$oft uses Quake clannies to wage war on Iraq! - explodingheadboy
[ Parent ]
Heh, since we're on the topic.... (4.00 / 1) (#96)
by Miniluv on Mon Dec 17, 2001 at 12:10:36 PM EST

Here's my experience with SMP on multiple x86 operating systems:

Solaris x86 blows at everything. It's Solaris Sparc with no good features and all the bad ones are even worse. Is it obvious yet that I'm not a Solaris fan?

Linux SMP is decent, but it doesn't scale spectacularly well just yet. They're working on it, and as far as free *nix goes, it's pretty damn good. Based on what I've read on LKML it seems they've still got highmem issues, which isn't unexpected, as x86 memory addressing for large memory configurations is a cast-iron bitch.

FreeBSD Stable/Release SMP is pretty good, scales almost as well as Linux. I've not heard anything one way or the other about large memory configurations, so I have to plead ignorance there. As far as current goes, the SMPng stuff they're working on sounds interesting, but take a look at their release schedule to see why they think they can accomplish so much. We aren't going to see 5.0 as a release until at least the end of 2002, more likely early in 2003 (they've slipped the date by at least 18 months now) and they aren't backporting any of the SMPng stuff.

The one thing FreeBSD has going for it in a huge way is the VM, which is arguably still better than Linux post 2.4.10 (with the Arcangeli VM) including the use once fix that's in 2.4.17-rc1. I've broken several boxes in my lab at work by doing simple things which hit file system caching in a huge way (think "grep -r stuff /") and suddenly memory starvation becomes a huge issue and the box starts oopsing.

If anyone in the K5 management crew is listening, I'd stick with Linux for the time being, as switching to FreeBSD would be more trouble than it's worth at this point.

Applied Solipsism worked for me.
[ Parent ]

Maybe... (none / 0) (#98)
by trhurler on Mon Dec 17, 2001 at 12:27:57 PM EST

It might be because FreeBSD's SMP and threading aren't up to snuff. Or maybe it is because it might actually require porting a few things, such as, say, cluster management software, which FreeBSD just doesn't have.

Then again, maybe it is because people who use UPSes as God Himself[tm] obviously intended don't have filesystem problems.

A third possible reason is that FreeBSD has this interesting habit of working only on a (sizable, but largely unstated) subset of hardware. It tends to do interesting things like flat out bomb on other hardware, often because of little hardware bugs that the Linux people wrote workarounds for long ago. Size does matter in a development effort, sometimes. I've convinced FreeBSD to crash for no good reason on at least three different machines around the office, and that's recent versions. Older ones were worse.

Yet another reason may well be "they like Linux." That's a pretty good one, considering that they run the site and you don't.

By the way, I run OpenBSD, and I regard your FreeBSD snobbery as beneath contempt. :)

'God dammit, your posts make me hard.' --LilDebbie

[ Parent ]
What I did when K5 was down / problems (none / 0) (#72)
by p0ppe on Sun Dec 16, 2001 at 03:27:40 PM EST

I actually began my career as a karma slut over at The other site.

Regarding K5, is it just me, or is K5 actually slower now, than before the downtime?

"Democracy is three wolves and a sheep voting on what to have for dinner."
I've noticed that too (none / 0) (#74)
by Teribaen on Sun Dec 16, 2001 at 04:48:10 PM EST

Regarding K5, is it just me, or is K5 actually slower now, than before the downtime?

Not that it was zippy before, but it does seem to be a bit slower.

[ Parent ]

Should be better now (none / 0) (#78)
by rusty on Sun Dec 16, 2001 at 09:03:10 PM EST

The initial re-launch was on 2/3 of the hardware we planned on. This morning, another of those two went down, so for a while we were on one machine, roughly the same size as hex was. A giant leap backward.

All three machines are up now, and we're going to be adding more. So it'll speed up in the near future. Meanwhile, right now we ought to be able to handle more traffic overall than bubba could. We're still tuning things though.

Not the real rusty
[ Parent ]

performance vs. scalibility (none / 0) (#79)
by rampy on Sun Dec 16, 2001 at 09:43:43 PM EST

FWIW there is a difference between performance and scalibility. Although manager Pointy Haired Boss types usually mean both when asking for either. =(

www.randomdrivel.com -- Fish, plankton, sea-greens, and protein from the sea!
[ Parent ]
Looking Forward to Rusty's Credit Card findings... (5.00 / 3) (#80)
by rampy on Sun Dec 16, 2001 at 09:57:10 PM EST

I'm looking forward to hearing what Rusty has decided/learned about credit card processing. hint, hint... nudge nudge... don't forget to post your findings, man...=P

www.randomdrivel.com -- Fish, plankton, sea-greens, and protein from the sea!
OH OH OH ME TOO ME TOO (none / 0) (#99)
by CmdrPinkTaco on Mon Dec 17, 2001 at 12:39:02 PM EST

I just had to write an HTTP Client in Java so that we can do out CC/ACH handling and am interested to see if others had some of the same problems that I have had. Im not so much interested in the process, having been through it already, but would like to see what others came up with.

As an aside, if anyone is interesed in the source for a command line java SSH HTTP Client (wow, that's a lot of buzz words) e-mail me at emericle@nospam.hotmail.com and I will send it to you - it isn't 100% complete yet but should be by the end of the week. Oh, and it's based off of www.innovation.ch/java/HTTPClient/ (site currently appears to be down, but it was up last week) which is GPL'd, which means mine is too. It's the best virus that I have ever had :)
Guess CmdrPinkTaco's .sig and win - nothing :)
[ Parent ]
Gee, I am glad K5 is back... (none / 0) (#89)
by DavidatEeyore on Mon Dec 17, 2001 at 04:30:24 AM EST

while you were off the air, i read the the daily-ish bulletins, rebuilt an old P166 and put linux on it, and started to rebuild my Win2K box which had gotten very full of odd bits of code, web crud, scripts, patches etc. It (the Win2k box) is now nice and clean again.. Glad to see you back again.

Btw. (none / 0) (#92)
by gromgull on Mon Dec 17, 2001 at 08:49:57 AM EST

thanks for Acts of Gord

Kept me "busy" at work for at least 3 days.
If I had my way I'd have all of you shot

Sweet. (none / 0) (#97)
by misterluke on Mon Dec 17, 2001 at 12:16:17 PM EST

Monday morning, and my favourite timewasting device is up and running again ( and pretty fast, too ). Thanks to everyone who got it back up and will hopefully keep it that way. Cheers.

Somehow, (none / 0) (#100)
by greyrat on Mon Dec 17, 2001 at 03:23:26 PM EST

the logo dosen't seem out of place anymore...
~ ~ ~
Did I actually read the article? No. No I didn't.
"Watch out for me nobbystyles, Gromit!"

Thank God You've Come Back Rusty ! (none / 0) (#102)
by Inden on Mon Dec 17, 2001 at 08:40:37 PM EST

My sojourn in the "Fountainhead" lands of the arch "no trolling ;-)" writing style tightasses of the "you'll never be good enough" adequacy culture was good for my discipline and for that I give them a lot of credit.

But they are hard hard hard to like. They don't seek affection, they seek perfection, I suppose. I like the graphically much warmer colors of the k5 screen too.

Please don't go away like that again. If you do, please make sure you take that bunch of Romans over at adequacy with you okay????
Libertarianism is Anarchism for the Rich
Looks like I picked a good time to come back (none / 0) (#104)
by GlennC on Wed Dec 19, 2001 at 12:00:58 PM EST

After I started my new job and got settled in, I decided that I'd see what was happening on K5. I stopped going after seeing all the 9/11 related postings, and figured that it would have calmed down a bit.

Then I hit "Operation Enduring Downtime".

I also read about what happened to Lee (I hope you're all right, buddy).

Anyway, Yet Another Lurker is back and ready to read again. Thanks to Rusty, Inoshiro, el guapo, the folks at Voxel, and anyone else who had a hand in bringing K5 back up.

That's my take on it. Take it for what you think it's worth.
Well. That Sucked. | 105 comments (105 topical, 0 editorial, 0 hidden)
Display: Sort:


All trademarks and copyrights on this page are owned by their respective companies. The Rest © 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!