Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

[P]
Reaching My Limits

By skyknight in Meta
Wed May 25, 2005 at 11:50:51 AM EST
Tags: Kuro5hin.org (all tags)
Kuro5hin.org

I have been a "full membership" subscriber to K5, if my memory serves, for about two or three years now, and a member of the community for even longer. My subscription ran out yesterday and I'm debating whether it makes sense to renew it.


When I first started paying for membership, I would sign up for six months at a go. I had no reason to fret over the long term viability of K5. The only cap on my willingness to sign up for blocks of time was the result of my not seeing the need to have that chunk of money in someone else's pocket before I was getting its benefit. More recently, in light of the shaky state of affairs here, I have been inclined to sign up for only three months at a go, and the last few times for only a single month. Now, with that last payment having been expended, I am wondering whether I ought to renew.

It's not about the money. I pay $5 for an Ultimate Double Whopper that is gone in a couple of minutes, so $4 for a month of K5 is no big deal, that is, if I feel like I'm actually getting something of value for it. I like K5. There are several people here with whom I very much enjoy interacting, the stories can be quite good, and despite its tending toward being a ghetto, even the diary section has its moments. Reply notification and spell check are a nice compensation for my $4/month, but they're not the real reason I fork over the cash. I am an able Perl programmer who knows his way around databases, and it would be trivial for me to do some screen scraping and caching that would match, nay, exceed, K5's functionality. I am perfectly happy writing my stories and comments in Emacs, running ispell, and then dropping things into the text box. The real reason I send Rusty $4/month is that I like to pay for the things that I enjoy. I know he is giving it away for free, but I don't care.

Forget about story quality, troll prevalence, and all that... Those are quantities that ebb and flow with time. I just want K5 to work from a technical standpoint. I'd really like for there to be a proper search feature here, but I'd also like a pony and a million dollars, so let's be reasonable... I'd settle for good page load times. I pay $20/month to subscribe to the WSJ, and this gives me access to their web site. Their typical page load time is ~1 sec. If it were common for my connections to them to hang for two minutes and then time out, you can bet I wouldn't shell out $20/month for that, yet this is precisely the kind of aggravating treatment I get at K5. This totally ruins the whole "K5 Visit" experience.

I fear that this phenomenon evinces an autocatalytic downward spiral, in which user disappointment in page load time (among other things) causes exodus from the site, and a diminished user base results in administrators feeling less and less motivation to keep things running smoothly, each reinforcing the other. I'm not just griping about one-off behavior that I've seen. The degradation of page load time that I lament is a trend I have seen continuing over the course of months.

Worse still, it's not just users, paying or otherwise, that get hurt. Advertisers suffer as well. When you click through an advertisement on K5, you are first directed to a form at K5 that processes the ad impression, and then forwarded to the advertiser's site. When K5 is bogged down, engaging in its "do nothing for two minutes and then hangup" shenanigans, this causes advertisers to lose traffic to their site. Yes, if you're technically savvy you'll note that the advertiser's web page URL is embedded in the K5 ad URL (a security flaw akin to the shopping carts of yore that stored product prices client side), allowing you to rip it out and visit the site directly, but then K5 loses ad revenue because the impression is not recorded. All in all, adequate page load times are essential to K5's viability, and yet they are wholly inadequate.

Furthermore, I should think that these problems are the kind of thing that some simple monitoring scripts would resolve. Why can't there be a trivial off-site script that does nothing more than load K5's front page every thirty seconds or so and send an alarm to administrators if the load time exceeds an acceptable threshold? Upon receipt of an alarm, an administrator could diagnose the situation, apply a manual resolution in the short term, implement an automated fix for the medium term, and start a discussion on how to fix the underlying problem in the long term. Is any such alarm and issue resolution system in place? If not, then do administrators care enough to bootstrap one? I think it would be a huge boon for the community to have such a thing. I would even help.

In the end, my fretting over whether to renew my membership was for naught:

You are ordering 1 month of Full Membership, for a total cost of $4.00.
The following errors were found in your order form:
An error occurred when I tried to process this transaction.
The credit card processor said: CC-2501: Unable to connect to SSL Server.
This is order number .

Sponsors

Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure

Login

Poll
K5 Page Load Times:
o Acceptable 85%
o Unacceptable 14%

Votes: 115
Results | Other Polls

Related Links
o Also by skyknight


Display: Sort:
Reaching My Limits | 137 comments (129 topical, 8 editorial, 0 hidden)
You want a pony? (1.85 / 7) (#1)
by mostes on Tue May 24, 2005 at 10:36:01 AM EST

But why? Ponies smell and their upkeep costs shitloads.

Not to mention you can't actually ride them unless you're a little girl.

PLEASe SuBMIT tHAT As A STORy! (1.80 / 5) (#7)
by NoMoreNicksLeft on Tue May 24, 2005 at 01:19:04 PM EST

Little girls riding baloney ponies... I'll be back in 20 minutes. Ok, ok, 2 minutes. We'll talk more then..

--
Do not look directly into laser with remaining good eye.
[ Parent ]
"their upkeep costs shitloads" (none / 1) (#99)
by BigZaphod on Thu May 26, 2005 at 04:43:12 PM EST

.. not to mention the loads of shit they tend to produce...


"We're all patients, there are no doctors, our meds ran out a long time ago and nobody loves us." - skyknight
[ Parent ]
IAWTP (none / 1) (#6)
by undermyne on Tue May 24, 2005 at 11:49:02 AM EST

+1 FP. No brainer.

ROR at the CC processing error. It pleases me greatly that Rusty's ability to collect money performs just as well as K5 on the whole.


"Coffee makes me go poop." thekubrix
Rusty says (none / 1) (#8)
by wre on Tue May 24, 2005 at 01:59:43 PM EST

He deliberately disabled the comment search because it was too easy to flood (and thereby take down K5). But of course, why doesn't he just enable it for the paid members?

Actually... (none / 1) (#9)
by skyknight on Tue May 24, 2005 at 02:13:25 PM EST

I was one of the people who encouraged him to turn it off originally. I am aware of how it was trashing things as a result of its inefficient implementation. To work properly, it would need intelligent indexing, and as far as I know it was originally implemented in a brute force fashion.

Today, though, I don't even really care about comment searching. It is turned off and K5 is still unbearably slow. For whatever reason, K5 is effectively broken when it comes to the operation of its servers. I can only just barely stand to use it. My usage habit is at present to load a page and then go do something else. Maybe when I come back it will have loaded, or maybe my connection will have timed out. This is not acceptable.



It's not much fun at the top. I envy the common people, their hearty meals and Bruce Springsteen and voting. --SIGNOR SPAGHETTI
[ Parent ]
He disabled payment for the same reason (none / 1) (#10)
by nkyad on Tue May 24, 2005 at 02:37:40 PM EST

It is too easy to flood (and thereby take up K5) and he has no desire to get rich.

Don't believe in anything you can't see, smell, touch or at the very least infer from a good particle accelerator run


[ Parent ]
K5's Overture (none / 1) (#11)
by nkyad on Tue May 24, 2005 at 02:51:58 PM EST

History recalls how great the site can be
While everybody's sleeping, the servers died out on him
Borne on the wings of time
It seemed the answers were so easy to find
To late, the profit's nigh
K5's sinking, let's take to Mefi

not that Supertramp is all that great, but I have just listened this song on the car...

Don't believe in anything you can't see, smell, touch or at the very least infer from a good particle accelerator run


Ultimate Double Whopper. (2.00 / 4) (#12)
by Nosf3ratu on Tue May 24, 2005 at 03:17:07 PM EST

Sweet Jesus, you've actually eaten that thing?

And you're still walking around and talking? You're a stronger man than I.


Woo!

Yes, but... (none / 0) (#15)
by skyknight on Tue May 24, 2005 at 03:44:04 PM EST

I've never accompanied it with fries and a soda. You see, I'd like to live past 30, and having 2500 calories in a single sitting is probably not a good way to accomplish said goal.

It's not much fun at the top. I envy the common people, their hearty meals and Bruce Springsteen and voting. --SIGNOR SPAGHETTI
[ Parent ]
YEAH BECAUSE FRIES HAVE CARBS (2.00 / 3) (#21)
by Nosf3ratu on Tue May 24, 2005 at 04:59:52 PM EST

YOU ARE THE PICTURE OF HEALTH LAWL.

p.s. let's make a baby.


Woo!
[ Parent ]

It is bliss (none / 1) (#18)
by LilDebbie on Tue May 24, 2005 at 04:24:59 PM EST

1200 calories of bliss. It settled in my stomach the way I imagine a brick would.

My name is LilDebbie and I have a garden.
- hugin -

[ Parent ]
Perl Programmer Eh? (1.66 / 3) (#13)
by unknownlamer on Tue May 24, 2005 at 03:30:29 PM EST

Submit patches to scoop to add the features you want.

Otherwise don't bitch.



--
<vladl> I am reading the making of the atomic bong - modern science
Um... (2.87 / 8) (#14)
by skyknight on Tue May 24, 2005 at 03:40:24 PM EST

You realize that running K5 involves a lot more than Perl programming, right? You must also realize that much of the parameters of how K5 is administered is not public knowledge. Lastly, you must realize that K5 is hardly ever running reasonably up to date Scoop code, and thus any work I do on the Scoop code base is unlikely to get installed on K5 on a timeline that I might consider reasonable.

Clearly you do not realize this, or you would not make such asinine comments. Furthermore, I previously volunteered to write some monitoring scripts for K5, Rusty said to do so and mail them to him, I did, and he proceeded to forget about it and ignore me whenever I brought the matter up again in the future.

So, bitch I shall, bitch. :-)



It's not much fun at the top. I envy the common people, their hearty meals and Bruce Springsteen and voting. --SIGNOR SPAGHETTI
[ Parent ]
Well (none / 0) (#22)
by unknownlamer on Tue May 24, 2005 at 05:01:28 PM EST

rusty recently said that upgrading scoop on k5 was going to happen soon.



--
<vladl> I am reading the making of the atomic bong - modern science
[ Parent ]
Well... (none / 0) (#24)
by BJH on Tue May 24, 2005 at 05:41:23 PM EST

...that could be true, I guess, but if I were you I wouldn't hold my breath.
--
Roses are red, violets are blue.
I'm schizophrenic, and so am I.
-- Oscar Levant

[ Parent ]
for some definition thereof. (none / 0) (#25)
by aphrael on Tue May 24, 2005 at 05:45:52 PM EST



[ Parent ]
Ha. (none / 0) (#26)
by skyknight on Tue May 24, 2005 at 05:55:46 PM EST

Would that definition put us before or after the universe has run its course, leaving the Earth a cold and dead ball of rock hurtling lonesomely through space? :-)

It's not much fun at the top. I envy the common people, their hearty meals and Bruce Springsteen and voting. --SIGNOR SPAGHETTI
[ Parent ]
Either is an upgrade. (none / 0) (#83)
by glor on Wed May 25, 2005 at 08:24:59 PM EST


--
Disclaimer: I am not the most intelligent kuron.
[ Parent ]

Yeah (3.00 / 2) (#29)
by Spendocrat on Tue May 24, 2005 at 07:47:16 PM EST

Right after the CMF gets up and running.

[ Parent ]
It was going to be 'this weekend' some months agox (none / 0) (#35)
by jongleur on Tue May 24, 2005 at 10:17:45 PM EST


--
"If you can't imagine a better way let silence bury you" - Midnight Oil
[ Parent ]
Tho I don't care, for me it's the content (none / 0) (#37)
by jongleur on Tue May 24, 2005 at 10:37:18 PM EST

more than anything. I would like a spiffier site, sure, but if it's to be upgraded I'd really like to see something next-generation. I feel real reorganization is possible though I don't know what I'd do.
--
"If you can't imagine a better way let silence bury you" - Midnight Oil
[ Parent ]
I know what I'd do ;-) (none / 0) (#56)
by rusty on Wed May 25, 2005 at 09:57:50 AM EST

Seriously. It's progressing. It'll be this summer some time. I'm not even about to try to be any more specific than that though. :-)

____
Not the real rusty
[ Parent ]
Lastly, (2.50 / 2) (#34)
by Reynolds Number on Tue May 24, 2005 at 09:51:32 PM EST

You do not realize that people like Hulver have already written fully-functional Perl code to fix things like search and submitted it as a patch, but Rusty refuses to implement it.

[ Parent ]
That's because ... (none / 0) (#43)
by Peahippo on Wed May 25, 2005 at 01:59:56 AM EST

... of all the lines with peppered with "53KR1T_B4CKD00R" and "RU5TY_34T5_1T". You know: Hulvercode.


[ Parent ]
This is a reason I never bothered to work on scoop (none / 0) (#103)
by SoupIsGoodFood on Fri May 27, 2005 at 02:02:34 AM EST

I was planning to clean up the HTML, so that K5 would be running on nice, complient HTML 4.01 and CSS. It would not only look better but I'm sure rusty would like the reduced bandwidth required for each page.

Then I had plans to replace the current, buggy auto-format and plan-text filters with Markdown, and maybe even dive into the Perl and fix up some of the idiosyncracies like the one option polls etc.

Then I realised that it's not worth the effort. The impression I got was that my help was simply not wanted, and that nobody really gave a fuck about Scoop's current state. Esspecially for K5.

[ Parent ]

If you want to foster a developer community... (3.00 / 2) (#106)
by skyknight on Fri May 27, 2005 at 07:05:34 AM EST

then good code modularization is a huge boon. The more loosely coupled the various components of a system, the less a new developer has to learn in order to contribute, and thus the less intimidating that first foray will, and the more quickly he can can see ROI for his labors. I've actually spent some time reading through Scoop source code, and listened to other people describe it, and clearly it does not exhibit loose coupling. This is a shame. Rusty deserves substantial kudos for many things, but careful and deliberate engineering is not one of them.

It's not much fun at the top. I envy the common people, their hearty meals and Bruce Springsteen and voting. --SIGNOR SPAGHETTI
[ Parent ]
careful planning (none / 0) (#111)
by janra on Fri May 27, 2005 at 04:07:09 PM EST

Well, the plan for scoop 2 is that it'll actually have one... ;-)

Seriously though, one of the big changes that's planned for scoop 2 is that it'll be very modular - right now it's quite monolithic, and you have to have ads, the calendar, everything loaded up even if you don't use any of them. The idea is to have nearly everything structured as a plugin, even core stuff like stories and comments.
--
Discuss the art and craft of writing
That's the problem with world domination... Nobody is willing to wait for it anymore, work slowly towards it, drink more and enjoy the ride more.
[ Parent ]

What exactly is the plan for Scoop 2? (none / 0) (#112)
by skyknight on Fri May 27, 2005 at 05:14:23 PM EST

I know nothing about it. Is it going to be a re-write from scratch? Is it already under way, or just in a planning stage at present? I might be interested in contributing depending on how those questions are answered.

It's not much fun at the top. I envy the common people, their hearty meals and Bruce Springsteen and voting. --SIGNOR SPAGHETTI
[ Parent ]
The plan isn't exact yet (none / 0) (#114)
by janra on Fri May 27, 2005 at 05:37:32 PM EST

Is it going to be a re-write from scratch?

Parts will be lifted from the current Scoop, but the structure will be significantly different. (More modular, as I mentioned, for one.)

Is it already under way, or just in a planning stage at present?

It's in the early planning stages at the moment. Once we get 1.1 released, we're going to work on planning it in more detail. Right now it's just a collection of notes for things we want to do with it. I'm itching to get started on it, but we can't just abandon 1.1...

I might be interested in contributing depending on how those questions are answered.

Cool :-)
--
Discuss the art and craft of writing
That's the problem with world domination... Nobody is willing to wait for it anymore, work slowly towards it, drink more and enjoy the ride more.
[ Parent ]

Alright, well, in that case... (none / 0) (#115)
by skyknight on Fri May 27, 2005 at 05:52:11 PM EST

how does one get involved? Where is all this planning and discussion taking place? Is there an ongoing forum on it somewhere, or is it just a collection of threads at scoop.kuro5hin.org?

It's not much fun at the top. I envy the common people, their hearty meals and Bruce Springsteen and voting. --SIGNOR SPAGHETTI
[ Parent ]
partly in #scoop (none / 0) (#116)
by janra on Fri May 27, 2005 at 06:09:27 PM EST

where all the developers idle...

Believe it or not, some planning actually does take place there ;-)

I also posted a poll on scoop.k5 asking for input on scoop 2, in general terms, and that got a couple of responses. That's more of a priorities wishlist though.

And you get involved by going to #scoop and participating in the discussions.
--
Discuss the art and craft of writing
That's the problem with world domination... Nobody is willing to wait for it anymore, work slowly towards it, drink more and enjoy the ride more.
[ Parent ]

If this is going to be more of a planned effort... (none / 0) (#117)
by skyknight on Fri May 27, 2005 at 06:22:33 PM EST

wouldn't it be good if the end result of all these discussions were a functional spec and a design spec? Why not set up a CVS/SVN repository somewhere, lay out a nice document hierarchy, and evolve the documents over time to reflect all of the arguments that contributors have and decisions that are made? That way, at any point in time a newcomer could go and look at the present state of affairs, bring [him|her]self up to speed, and begin contributing shortly thereafter. Short of that, what am I, someone who has thus far been out of the loop, supposed to do? It would be a real nuisance to have to have every single new person pester older members for information. There ought to be something they can read. Furthermore, this would serve to disambiguate things. Two developers might have a different idea of how things are to work, and a written document is the only practical way to resolve the dispute without a town hall meeting with a quorum of developers present.

It's not much fun at the top. I envy the common people, their hearty meals and Bruce Springsteen and voting. --SIGNOR SPAGHETTI
[ Parent ]
specs and docs (none / 0) (#118)
by janra on Fri May 27, 2005 at 07:03:41 PM EST

That will be part of the process, but we're using a wiki instead of cvs. When it's developed enough I may turn it into a more formal document, but the wiki lets us cross-reference things and build the structure easily.
--
Discuss the art and craft of writing
That's the problem with world domination... Nobody is willing to wait for it anymore, work slowly towards it, drink more and enjoy the ride more.
[ Parent ]
Cool. (none / 0) (#120)
by skyknight on Fri May 27, 2005 at 08:04:46 PM EST

Does said wiki already exist? If so, then is it accessible to the public?

It's not much fun at the top. I envy the common people, their hearty meals and Bruce Springsteen and voting. --SIGNOR SPAGHETTI
[ Parent ]
#Scoop (none / 0) (#124)
by pHatidic on Sat May 28, 2005 at 09:53:07 PM EST

Why isn't #Scoop on freenet where most of the other open source dev channels are hosted? While my IRC client can technically connect to multiple IRC servers at once, it tends to crash randomly after several days.

[ Parent ]
beats me (none / 0) (#125)
by janra on Sat May 28, 2005 at 11:35:59 PM EST

My guess is that it's because #kuro5hin was on slashnet, and they eventually needed to split Scoop discussion from general chatting and just kept it on the same IRC network because it was easier for them.
--
Discuss the art and craft of writing
That's the problem with world domination... Nobody is willing to wait for it anymore, work slowly towards it, drink more and enjoy the ride more.
[ Parent ]
As some understand slowness to be... (2.00 / 2) (#16)
by killmepleez on Tue May 24, 2005 at 04:17:20 PM EST

I guess YMMV... the only thing that's slow for me is any search-and-retrieve function such as viewing a particular users recent comments. Otherwise, modding, reading stories, loading diaries, etc. only takes a second or two. It used to be absolutely dismal across the board, but since the last round of system updates [in, what, early 2004?] it's been pretty speedy. It still takes less time to render than monsters such as any page hosted by MSN.

__
"I instantly realized that everything in my life that I thought was unfixable was totally fixable - except for having just jumped."
--from "J
There is a ton of jitter... (none / 1) (#19)
by skyknight on Tue May 24, 2005 at 04:29:40 PM EST

Sometimes it works perfectly. Other times, it just stalls out completely. It's not just occasionally that this happens, either. This is a daily affair for me. I'll go to load a page and it will just grind incessantly, sometimes taking over a minute, sometimes just giving up altogether. Also, as Rusty suggested previously, it can't be because of comment reply notification cache building, because it's happening to me today and I'm no longer a subscriber, as of yesterday.

It's not much fun at the top. I envy the common people, their hearty meals and Bruce Springsteen and voting. --SIGNOR SPAGHETTI
[ Parent ]
That's very weird (none / 0) (#58)
by rusty on Wed May 25, 2005 at 10:01:22 AM EST

I'm here every day, and I've seen none of that. The only time it's slow is when it's recaching my replies.

I don't know how to explain that.

____
Not the real rusty
[ Parent ]

Hm... Maybe it's this chunk of code... (none / 0) (#60)
by skyknight on Wed May 25, 2005 at 10:22:12 AM EST

if ($user->get_handle() eq 'skyknight') {
    fuck_over($user);
}
elsif ($user->get_handle() eq 'rusty') {
    pamper($user);
}

I'm never one to discount that scenario. :-)

How often does reply re-caching occur, which is to say, how often does the code deem them to be stale and precisely what does that trigger?  Maybe I'm too long and thoughtful in some of my replies and K5 invalidates my cache whilst I toil over my prose?

It's not much fun at the top. I envy the common people, their hearty meals and Bruce Springsteen and voting. --SIGNOR SPAGHETTI
[ Parent ]

Heh (none / 0) (#62)
by rusty on Wed May 25, 2005 at 10:36:22 AM EST

There is actually one place where if the user name is rusty, it skips doing something that it would otherwise do. There's a box that shows who is currently an admin which only displays for admins. But I didn't like it always being there, so it has an extra check here and exempts me. :-)

Replies re-cache whenever they change, or after some arbitrary period of time. What happens is mysql actually caches the query for some period of time. I don't know in great detail when and how it decides it's not valid anymore, but epirically it seems to persist until you haven't hit the page for 15 or 20 minutes.

____
Not the real rusty
[ Parent ]

you're here eveyr day? (none / 0) (#107)
by dimaq on Fri May 27, 2005 at 08:31:01 AM EST

honest?

wow!

[ Parent ]

Really (none / 0) (#109)
by rusty on Fri May 27, 2005 at 11:28:44 AM EST

Well, almost. But if I'm in front of a computer at any point during a given day, it would be overcoming five years of daily habit not to hit K5 first.

____
Not the real rusty
[ Parent ]
believable explanation (none / 0) (#135)
by dimaq on Wed Jun 01, 2005 at 05:51:43 AM EST

the addiction that is :)

man you sowed this evil disease in thousands of good people! *g*

[ Parent ]

yeah, ymmv (none / 0) (#98)
by dke on Thu May 26, 2005 at 01:46:19 PM EST

Really.. maybe it's just me.. but k5 hardly ever seems to lag.. except in those moments of utter brokenness.. ;-)

Now, a really broken site.. orkut :S
Nothing is ever easy
[ Parent ]

Breathless with Suspense.. (2.50 / 2) (#17)
by Pluto on Tue May 24, 2005 at 04:20:51 PM EST

Did you actually reach your limit?

I've just returned after a 1.5 year hiatus, and I find things quite satisfactory and am grateful to be back.

For one thing, the load times seem improved to me. And the site has not gone toes-up for quite a white.

Furthermore, I actually approve of the quality of what I see now -- in context, of course to the devolution of the entire internet.

It would be a shame if you took yourself off, such a fine contributor that you are...

_______________________________________
Burgeoning technologies require outlaw zones... deliberately unsupervised playgrounds for technology itself. -- William Gibson

I don't plan on leaving... (none / 0) (#20)
by skyknight on Tue May 24, 2005 at 04:36:28 PM EST

but I'm sure that the frustration that I experience effects my posting habits substantially. There are times that I want to surf on and post to K5 and I just get up and walk away from my computer because the response time is hopeless. There are plenty of times when it is as smooth as the proverbial baby's bottom, but often times it is unbearable, and by "often times" I mean "on a daily basis".

It's not much fun at the top. I envy the common people, their hearty meals and Bruce Springsteen and voting. --SIGNOR SPAGHETTI
[ Parent ]
Related note (1.50 / 2) (#23)
by trane on Tue May 24, 2005 at 05:14:49 PM EST

why does it take so long to log out of the site? Isn't it a simple matter of (in java) Session.invalidate()?

+1FP skynight and k5 meta-bashing (1.00 / 2) (#27)
by Resonant on Tue May 24, 2005 at 07:01:58 PM EST

Entertaing post with good points, although I guess I have less problems with lagout than skynight does.

"I answer, 'This is _quantitative_ religious studies.'" - glor
Was there really any need for that rant? (2.00 / 5) (#28)
by I Hate Yanks on Tue May 24, 2005 at 07:47:06 PM EST

A simple diary entitled "K5 is dying [n/t]" would have achieved the same purpose.


Reasons to hate Americans (No. 812): Circletimessquare lives there.

Actually, I started it as a diary... (none / 0) (#30)
by skyknight on Tue May 24, 2005 at 07:56:11 PM EST

but decided that I wanted a bigger soap box, and it seems as if I am going to be granted it.

It's not much fun at the top. I envy the common people, their hearty meals and Bruce Springsteen and voting. --SIGNOR SPAGHETTI
[ Parent ]
you just spent 15 mins of your life over $4 /NT (2.00 / 2) (#45)
by ccdotnet on Wed May 25, 2005 at 02:31:45 AM EST



[ Parent ]
I thought I made it clear in the piece... (none / 0) (#52)
by skyknight on Wed May 25, 2005 at 09:34:52 AM EST

that the $4 was irrelevant. Consequently, I suspect you did not perform admirably on the PSATs. I suggest you take them again, this time enrolling in a prep course beforehand.

It's not much fun at the top. I envy the common people, their hearty meals and Bruce Springsteen and voting. --SIGNOR SPAGHETTI
[ Parent ]
PSATS ? (none / 1) (#81)
by ccdotnet on Wed May 25, 2005 at 07:00:31 PM EST

Just can't see beyond those borders can you?

[ Parent ]
Perl is good, but... (2.66 / 6) (#31)
by jd on Tue May 24, 2005 at 08:08:15 PM EST

...I think re-implementing Scoop - or writing a better engine - would be better in a language that gave you either the raw horsepower of the box or enough parallelism that power could be found by other means. Oh, Perl is a great language, and will do 99.99% of everything anyone wants, but it's designed to be more of the Volvo of programming languages - lots of room, great hauling power - than it is the Ferrari Formula 1 or the Bullet Train.

Kuro5hin is an OK site, but I wouldn't rate it much beyond OK, because people from "rival" special interest groups do set out to play at "gang war". I've seen plenty of people vote -1, not because the article is no good, but because they -1 that category "on principle". Hey, there's nothing wrong with not liking things, but there needs to be a mechanism for filtering those out, and there needs to be a mechanism for penalizing vote abuse by those who do set out to break the system.

I don't know what the "solution" is - maybe "slashboxes" would work, as then you don't have to "suffer" Front Page stories on subjects you don't have an interest in.

Also, the categorization tends to be a little... primitive. A simple pick-list (which inevitably doesn't have exactly what anyone wants) is not really that great, although it is a good start.

Anyway, that's getting a bit off the subject, other than to say I'm not impressed with any of the engines out there, but they ALL have good features. If someone could just combine them...

The question of paying is a good one. Personally, I'd say that paying for a product that is either not maintained or maintained badly is not a good idea - at least, on an unlimited basis. As such, I don't think K5 subscriptions are a sound way to go. It would be better to have some sort of "bounty" system in an escrow account - donors put money into escrow and request certain results. If the results are achieved, the money is delivered.

The sorts of things that might work would be, say, some specified bugs being fixed, certain features added, a certain number of stories in a given area of interest making Front Page, etc. Things Rusty has a certain level of power to deliver.

The prices would then reflect this kind of system, which means you'd need fewer "core" subscribers to keep the system running, but that "core" would have a strong say over standards.

As it stands, you pay money but have no guarantee of anything back. That's true in software in general, and I don't like that. If you pay money, you have a vested interest in something being delivered, but the other side has no interest or stake at all in the matter.

Such a one-sided contract is something I cannot, in good concience, tell anyone to go for. Realistically, you can't avoid it, but that doesn't mean it is ethical or proper conduct. As such, I would urge any subscriber to push for some system - ANY system - that they would be willing to pay for that produces accountability and quality, up to what they ARE willing to pay for.

(Nothing for nothing, in the end, so if you want K5 to improve, you've got to give something to make that happen. Whether that is time, money, or whatever. But if you DO give it, there should be some guarantee that you're going to get something back that makes it worth the while, by some means.)

Um... (3.00 / 2) (#41)
by trhurler on Wed May 25, 2005 at 12:44:16 AM EST

Actually, the gain you get from going from perl to, say, C(and it doesn't get any faster than C,) is less than an order of magnitude, and that's theoretical. The real gain is probably less than half that. I'm not promoting Perl, as Perl is kind of a shitty language, but it does have features that are essentially perfectly matched to this sort of software. I more or less gave up on my idea for a new system after realizing that yes, I could do it better, faster, and so on, but that by the time I would get it done, hardware advances would mean scoop could be faster than my software would have been had it been complete the day I started wishing for it.

As for one sided contracts, think about it. All you're putting up is money. And not that much of it, really. On the other hand, the people who produce software put in years of their lives. The result is one sided? No shit, Sherlock.

--
'God dammit, your posts make me hard.' --LilDebbie

[ Parent ]
Depends on how you define faster (none / 1) (#47)
by jd on Wed May 25, 2005 at 03:44:23 AM EST

On a single thread, C is certainly fast. And for a reasonable amount of effort is probably the fastest, yes. However, a discussion board is likely to have many users in parallel. It is hard to get accurate figures, but online communities can number a few hundred simultaneous users.

The problem is that using a thread per user is often a Real Bad Idea when you get into the hundreds of users, especially as many are: (a) read-only, and (b) often accessing exactly the same data.

What you really want is to divide the threads by activity, rather than by user. C is not really very good at this, because there's really no good way to represent objects and applications. C is flat and monolithic. An activity-based system is heirarchical and potentially distributed.

For a distributable, heirarchical system, you're looking more at SISAL and Occam, but there aren't that many good compilers for those out there.

Another consideration is that a message system needs to serve all the users at a reasonable speed, so you're also wanting some real-time characteristics. You don't want one user to lock the system up, or even have the potential for doing so.

Finally, you don't want the crashing of a single unit of the software to take down everything. You want fault-tolerance. C is about as intolerant as you can get, and it can be a real problem to write handlers for every possible signal in every possible component. And then have a health monitor, to allow for untrappable signals, so you can restart something.

Really, you want something that makes it: (a) easy to handle real-time, exceptions and errors, and (b) makes it possible to handle this centrally, efficciently and as transparently as possible.

If you don't, then you have to rely on adding all the hooks in every place that might need them, no more and no less. Few coders claim to be perfect, so a library or language feature that deals with all this is a Very Good Idea.

For robust code, you're more likely to be looking at a more modern, probably high-level, language. Those tend to have good quality exception handling. If it is targetted at engineers, then it may well also take care of the real-time properties.

So, what you end up with is something like this:

For the raw, underlying servlets/service modules, you're wanting a highly parallelizing language that supports process mobility, low-cost communication and low-cost threading.

For the framework that actually drives all of this and connects the bits together, then you're probably looking at a mugh higher level language. Speed isn't important, as it doesn't actually do any "work" on data, it is purely managerial.

What you want is something that'll work just fine on a single box in your basement, or be extended to the sort of cluster room that systems like Slashot inhabit, WITHOUT having to change anything. Changes add the risk of bugs, so should be carried out to add features or debug, but never for structural changes.

(If the structure changes enough to force a change in the code, then there's a good chance you've had to make enough non-trivial modifications that it's basically a new program. That is a Bad Design.)

A Good Design should be scalable, no matter what the userbase and no matter what the change in hardware. It should be aware enough that it can provide the best service for the users given the number of users and the hardware available to it.

Smart software is good software. Dumb software, these days, is stupid.

[ Parent ]

Er (2.66 / 6) (#87)
by trhurler on Wed May 25, 2005 at 09:57:41 PM EST

On a single thread, C is certainly fast. And for a reasonable amount of effort is probably the fastest, yes. However, a discussion board is likely to have many users in parallel. It is hard to get accurate figures, but online communities can number a few hundred simultaneous users.
And there is no reason whatsoever why they can't be handled with a nearly zero overhead locking mechanism of incredible simplicity and a select based multiplexing mechanism. You may think "that's not sexy," but it works better for this sort of problem than anything else I've ever seen performance wise. If you have several processors, create one thread per and share the load across them, but have them all run the same code. The resulting performance will STOMP whatever language of the day you're thinking of, and you can hide the nasty bits behind interfaces of reasonable simplicity.
The problem is that using a thread per user is often a Real Bad Idea when you get into the hundreds of users, especially as many are: (a) read-only, and (b) often accessing exactly the same data.
This really doesn't matter. 500 threads on a modern machine is irrelevant. As for reading, use read/write locks. Simple. BUT, for maximal performance for LOTS of users, see above.
What you really want is to divide the threads by activity, rather than by user. C is not really very good at this, because there's really no good way to represent objects and applications.
It is easy to represent objects in C. It just requires that you know what you're doing.
C is flat and monolithic. An activity-based system is heirarchical and potentially distributed.
C is precisely as flat and monolithic as C++. Perhaps what you mean is "a typical C program is flat and monolithic." That's true, but also irrelevant.
For a distributable, heirarchical system, you're looking more at SISAL and Occam, but there aren't that many good compilers for those out there.
And I would never ever use them anyway, for several reasons. First, since there's nearly nobody who knows those languages, writing in them means that if you go away, the software is likely worthless. Second, portability(real portability - not halfway there kind of sort of.) Third, they're languages of the week designed by wonks to address domain specific issues, and such languages are never well thought out for real world use. Fourth, it is easy enough to gain these capabilities in mainstream languages(in fact, this domain is a primary application of both Java and the various .NET tools.)
Another consideration is that a message system needs to serve all the users at a reasonable speed, so you're also wanting some real-time characteristics.
Maybe if you want to run it on a 386. Get a real computer.
You don't want one user to lock the system up, or even have the potential for doing so.
Then you're fucked. No system on earth can guarantee that. If what you mean is you would like to minimize the chances, then adopt good coding practices, vet your developers, and write in a language that has automatic resource management.
Finally, you don't want the crashing of a single unit of the software to take down everything.
Actually, I want the crashing of a single unit of the software to be so rare as to be unworthy of my consideration given that the application is not exactly critical anyway.
You want fault-tolerance. C is about as intolerant as you can get, and it can be a real problem to write handlers for every possible signal in every possible component.
It is trivial to ignore all signals that aren't going to kill you anyway if you don't want to give them some special meaning. As for fault tolerance, it isn't hard to split a system across multiple executables, and it might be a good idea.
And then have a health monitor, to allow for untrappable signals, so you can restart something.
Whereas you think if one writes in SISAL or Occam, a signal 9 won't kill your program? Do you actually understand how signals work, by any chance?:) They're not a language mechanism...
Really, you want something that makes it: (a) easy to handle real-time, exceptions and errors, and (b) makes it possible to handle this centrally, efficciently and as transparently as possible.
So you want C on POSIX. Gotcha.
If you don't, then you have to rely on adding all the hooks in every place that might need them, no more and no less. Few coders claim to be perfect, so a library or language feature that deals with all this is a Very Good Idea.
True, and yet unexciting. I can't think of a real language that doesn't provide for this if you use it correctly, except maybe perl.
For robust code, you're more likely to be looking at a more modern, probably high-level, language. Those tend to have good quality exception handling. If it is targetted at engineers, then it may well also take care of the real-time properties.
Exception handling is not a quality feature. It is also utterly incompatible with real time features of any kind. If you don't understand why, then you need to go learn some more before telling people how it is.
So, what you end up with is something like this:
Ooh... software architecture from an academia wanker who probably has never actually used this stuff for anything bigger than a term project. I can't wait!
For the raw, underlying servlets/service modules, you're wanting a highly parallelizing language that supports process mobility, low-cost communication and low-cost threading.
If I wanted to serve the entire world population maybe. To paraphrase someone you probably have never heard of whose name I don't recall, when the specification says "make a cup of tea," you don't need to boil the ocean.
For the framework that actually drives all of this and connects the bits together, then you're probably looking at a mugh higher level language. Speed isn't important, as it doesn't actually do any "work" on data, it is purely managerial.
Of course, you could get one high level language that has libraries implementing the various features you want and just use it...
What you want is something that'll work just fine on a single box in your basement, or be extended to the sort of cluster room that systems like Slashot inhabit, WITHOUT having to change anything. Changes add the risk of bugs, so should be carried out to add features or debug, but never for structural changes.
So then what I need is slashcode. Hmm... that can't be right.
(If the structure changes enough to force a change in the code, then there's a good chance you've had to make enough non-trivial modifications that it's basically a new program. That is a Bad Design.)
Says the guy who likely hasn't written ten thousand lines of code in his whole life. If the requirements change, that can mean arbitrary changes to the codebase, and you cannot always forsee these requirement changes. Sorry to burst your bubble with a bit of real life there, Mr. Thesis Writer.
A Good Design should be scalable, no matter what the userbase and no matter what the change in hardware. It should be aware enough that it can provide the best service for the users given the number of users and the hardware available to it.
I'm eagerly waiting for you to use the words "grid computing." Come on, you know you want to!
Smart software is good software. Dumb software, these days, is stupid.
Um... your design did not address any of the details of the problem at hand. All you did was list off some technological capabilities that apparently cause you to cream your jeans when you think about them. Have you ever had a real job in your entire life?

--
'God dammit, your posts make me hard.' --LilDebbie

[ Parent ]
You're right... (3.00 / 2) (#89)
by rusty on Wed May 25, 2005 at 11:27:49 PM EST

...but when I need Scoop to safely land the space shuttle, I'm totally going to look into all that other stuff he was talking about.

____
Not the real rusty
[ Parent ]
Heh! (none / 0) (#92)
by jd on Thu May 26, 2005 at 03:18:16 AM EST

Having worked at NASA, I can tell you that I'd rather trust Scoop to land the Space Shuttle than anything they're using right now.

Purely - and I mean purely - for giggles, I'd be happy to see if I could port Scoop (or parts thereof) to Occam or SISAL, if you'd get a kick out of it, or even would be interested in seeing how exotic languages would tackle the same problems.

[ Parent ]

Yikes (none / 0) (#105)
by rusty on Fri May 27, 2005 at 02:53:28 AM EST

What about the vaunted triple-redundancy and super-thorough testing that NASA's so known for? I mean, you do have to admit that they haven't had a major didsaster in any manned program that's traceable back to faulty software. By comparison, Scoop often doesn't even bother to check for basic errors, because lots of times I just don't care that much. :-)

____
Not the real rusty
[ Parent ]
I have to say... (none / 1) (#119)
by jd on Fri May 27, 2005 at 07:42:23 PM EST

...I am impressed that there has only been one publicly-acknowledged software disaster on the Space Shuttle (three computers crashed, when trying to deorbit the Shuttle, in one of the early flights).

Their systems use .rhosts files, the safety briefing included such words of wisdom as "do not jog on the runway", and they generally don't test products before they buy them (which is why they got very upset with the Pentium bug, as they had upgraded a lot of systems without seeing if they'd actually work).

I worked on a bunch of aerospace engineering applications for them, and I can say with complete honesty that if there was any data validation going on, I didn't see it. One application I was working with started off at 10 megabytes. It needed to be compiled with compiler debugs on, as there were undiscovered buffer overflows which caused it to crash if the debug symbols weren't there to pad the code out.

After cleaning out the cruft, fixing the buffer overflows and ironing out all the other assorted bugs, I had the program down to 720K with debug symbols, 200K optimized.

ANY program that is 93% garbage (with the remaining 7% bug-ridden beyond belief) should not be used outside of Software Engineering classes on what not to do.

The program was actually used by a major manufacturer to build high-performance aircraft.

To this day, I am astonished that as much stays in the air as it does, and the company concerned (which shall remain nameless to save it embarrassment for the crud it is using) deserves considerable credit for actually building something that works and works well, given the tools it is using.

So, when I say I'd rather trust Scoop, I mean it.

[ Parent ]

Yes, but... (none / 0) (#121)
by skyknight on Fri May 27, 2005 at 08:08:19 PM EST

I'd rather K5 got search working as opposed to code for de-orbiting. Let's have some priorities, shall we?

It's not much fun at the top. I envy the common people, their hearty meals and Bruce Springsteen and voting. --SIGNOR SPAGHETTI
[ Parent ]
There's a difference? (none / 0) (#122)
by jd on Fri May 27, 2005 at 08:18:07 PM EST

De-orbiting is just a search algorithm applied to the ground. The only real difference is you don't want it to work too quickly.

[ Parent ]
Hm. I don't know about that. (none / 0) (#127)
by skyknight on Sun May 29, 2005 at 08:05:48 AM EST

That strikes me as a rather stretched analogy.

It's not much fun at the top. I envy the common people, their hearty meals and Bruce Springsteen and voting. --SIGNOR SPAGHETTI
[ Parent ]
Well, (none / 0) (#102)
by trhurler on Thu May 26, 2005 at 09:35:30 PM EST

Ok. I don't think I'd use a wank language that probably doesn't even have a million lines of operational code and is probably full of bugs in its runtime and standard library for something like that, but then, I'm not going to fly on your space shuttle, so why should I care?:)

--
'God dammit, your posts make me hard.' --LilDebbie

[ Parent ]
not to mention (none / 0) (#91)
by emmons on Thu May 26, 2005 at 12:58:28 AM EST

As rusty has already pointed out, scoop ain't the bottleneck. Mysql is.

Perhaps he could pontificate on the proper design and implementation of an RDBMS?

---
In the beginning the universe was created. This has made a lot of people angry and been widely regarded as a bad move.
-Douglas Adams

[ Parent ]

Er, no... (none / 1) (#95)
by skyknight on Thu May 26, 2005 at 07:30:42 AM EST

I don't think that lackings in the MySQL engine itself are the root of the problem. Rather, there are architectural decisions in how Scoop goes about using it that are problematic. A better table structure and clever background processing could yield a huge boon for performance. See this thread in which Rusty and I mull over this issue.

It's not much fun at the top. I envy the common people, their hearty meals and Bruce Springsteen and voting. --SIGNOR SPAGHETTI
[ Parent ]
Language choice (none / 0) (#110)
by foobarian on Fri May 27, 2005 at 03:29:53 PM EST

For the raw, underlying servlets/service modules, you're wanting a highly parallelizing language that supports process mobility, low-cost communication and low-cost threading.
Sounds like a good idea. Maybe something like SQL would fit the bill? Parallel- yes. Mobility- doesn't matter, queries don't run long. Threading- as fast as write locks allow it to be.
For the framework that actually drives all of this and connects the bits together, then you're probably looking at a mugh higher level language. Speed isn't important, as it doesn't actually do any "work" on data, it is purely managerial.
Another wise idea. Such high level language would be easier to program in too, a boon for a project like Scoop. I know, let's use Perl with MySQL! Oh, wait.

[ Parent ]
You realize... (none / 0) (#113)
by skyknight on Fri May 27, 2005 at 05:16:44 PM EST

that "SQL" is just a query language and that it doesn't actually say anything about the underlying DBMS, right?

It's not much fun at the top. I envy the common people, their hearty meals and Bruce Springsteen and voting. --SIGNOR SPAGHETTI
[ Parent ]
Minimizing programmer time (none / 0) (#55)
by rusty on Wed May 25, 2005 at 09:52:21 AM EST

I've always thought that software like this will inevitably be most successful when it's written in a language that tends to minimize programmer time (like perl or PHP) rather than processor time. Programmer time is by far the most scarce resource involved in producing a site like this. It may be 100% slower in perl, but it's trivial to dig up 100% more processor time. But if it's twice as hard to develop for it, you're kinda screwed.

____
Not the real rusty
[ Parent ]
Well, see, the thing is... (none / 1) (#63)
by skyknight on Wed May 25, 2005 at 10:39:58 AM EST

in my experience the output rate of the programmer is not just a function of the programming language and the programmer's skill in that language. Another factor is the complexity of the code, and some languages fair better than others in managing it. I've worked on Perl projects of all sizes, ranging from the single shot ten line scripts to the 10k+ line behemoths, and in my experience Perl just does not do well under the strain of the latter. There are huge productivity gains for using Perl for rapid prototyping, and for small to medium sized projects, but for large projects, which I'm sure is a reasonable category into which to slot Scoop, Perl tends to collapse under its own weight.

With a programming language like Java or C++, you're going to be moderately annoyed at the amount of effort required at the outset of the project to get little things done. However, when they are in fact done, they are apt to be more solid, if you're a decent practitioner of those languages. In Perl, I don't think that's the case. Being disciplined will help you, but if you're actually being really disciplined then you're going to be putting in as much effort as you would using C++ or Java, so why not make the jump? At least then your "discipline" would be of a standardized kind as opposed to your own flavor, thus making developer collaboration more difficult.

To me, Perl's biggest failing is its blatant adhockery when it comes to object orientation. Member data is stored as elements of a hash (typically, unless you want to make things even worse for yourself), and there is no conception of public/protected/private. This has myriad disastrous consequences. Having people poke at an object's internals is troublesome for maintainability, as it makes it exceedingly difficult to change them. Inheritance is incredibly sticky because all of the classes are typically sharing the same hash to store values. It's way too easy for multiple inheritance or deep inheritance to have different classes step on one another's toes. How can you be sure they wont? Whereas in something like C++ or Java the compiler takes care of all that for you, in Perl you would basically have to analyze every single line of code for the classes involved in the inheritance.

So, the moral of this story is that one must take great care not to focus on programmer productivity at any given stage of the project. Somehow, and I admit this is difficult, the real goal ought to be to maximize the sum of programmer productivity over the life time of the project. This is tricky because most successful software projects are accidental. Perhaps, then, the real trick is knowing when to transition from a rapid prototyping mode to a more serious development mode, possibly rewriting the code in a language more conducive to large scale software engineering endeavors.



It's not much fun at the top. I envy the common people, their hearty meals and Bruce Springsteen and voting. --SIGNOR SPAGHETTI
[ Parent ]
This is true (none / 1) (#64)
by rusty on Wed May 25, 2005 at 10:49:28 AM EST

However, Scoop's half-assed object orientation seems to mesh pretty well with perl's half-assed object orientation. There's (almost) only one object in Scoop, so everything pretty much has access to everything else all the time. OO purists would howl in pain and dismay to see it, but it has made it pretty easy for someone to jump in on one little feature and make some improvements without learning an entire object inheritance structure and API. It's basically object oriented to the extent that being so helps. There are a few drawbacks to this approach, but there's drawbacks to every approach.

Your point about when to transistion from rapid prototyping to more serious development is true though. Scoop has actually started to make that transition, in the development process and how things like patches and releases are managed. It's a lot more structured now than it used to be. The next major release may also change the code structure to be more OO where it will help to be, which would be an improvement.

But I can't really say we're likely to change languages. For one thing, I have very little interest in learning another language, or in rewriting everything in some other language and re-debugging it all over again. And I suspect everyone else who actually writes code for Scoop feels much the same way.

____
Not the real rusty
[ Parent ]

Well, it cuts both ways... (none / 1) (#67)
by skyknight on Wed May 25, 2005 at 11:40:27 AM EST

Having a low barrier to entry is great if you're trying to cultivate a developer community. On the other hand, having a high barrier to entry, while resulting perhaps in fewer contributors, makes the contributions that do happen be of (generally) higher quality as they will tend to be better considered. If anything goes, then over the years you're going to amass a motley crew of inconsistent idioms. Limiting your developers' options can actually be a powerful feature if done judiciously.

Furthermore, doesn't the relative lack of OO tend to expose a lot of the internals to developers. Might it not actually be easier for new developers to grok the system if, at least at first, they can think in terms of high level objects, and then delve further down into the details as necessary? Also, good encapsulation will leave developers less fearful that they are going to break something by misunderstanding the assumptions of the code. Having well defined interfaces, with explicit pre-conditions, post-conditions and effects can go a long way towards easing developer fears. Mind you, OO isn't the only way to go about this, but in my experience it tends to foster a way of thinking that is conducive to it.

I agree that at this point it would be really painful to change languages. It wouldn't just be a matter of you learning a whole new language, but rather the whole Scoop developer community. A decision to do as much on your part could very well lead to a project fork.

That being said, have you looked at Ruby? I just finished up with school last week and so have a bit more time on my hands and have been plowing through Programming Ruby, and it looks like an excellent language. It's built from the ground up to be a true OO language, and thus far I've yet to read something about it and say "yeah, but Perl does this, Ruby doesn't, and I couldn't live without it". In fact, Ruby was explicitly designed to be friendly to Perl programmers, as Perl was a major source of inspiration for it. For example, its pattern matchin is done via objects, but Ruby also puts the results of matches and such into special variables with the same names that Perl uses, thus making the transtion from Perl to Ruby very easy. In fact, it would seem that some of the less ugly Perlish idioms are preferred to the OO constructs. I think I could feel very at home in Ruby, and it would provide me with the OO constructs that Perl is so sadly lacking. I know you're a busy guy, but hey, who isn't? Do yourself a favor and pick up a copy of Programming Ruby.



It's not much fun at the top. I envy the common people, their hearty meals and Bruce Springsteen and voting. --SIGNOR SPAGHETTI
[ Parent ]
Developer community, and libraries (none / 1) (#70)
by rusty on Wed May 25, 2005 at 12:10:40 PM EST

At this point, Scoop still does have a very small developer community. Part of that is probably the fault of the system. Drupal tends to have more developers because people can write plugins without knowing very much about the whole system. Another part of it is probably that perl is its own barrier to entry, compared to php. Any idiot can learn some php, and it seems like most of them do. ;-)

But besides that, all I can really say is that Scoop has finally developed a pretty consistent idiom, and for the most part it works. I don't really know why. But it does. However, a way to write fairly distinct plugins without knowing a lot about the rest of the code would probably help us out quite a bit. We'll get there. Eventually.

The other problem with changing languages that I didn't even think of till now is libraries. There are a number of CPAN libs that Scoop relies on to do stuff. Changing languages would mean having to find good replacements or rewriting all of these as well.

____
Not the real rusty
[ Parent ]

<--- I'm with stupid (3.00 / 3) (#93)
by xL on Thu May 26, 2005 at 06:40:28 AM EST

Perl is a magical language. It allows for a "Stream of consciousness" style of approaching programming that has turned many a Unix nerd into its admirer. Like the bourne shell, perl is a large cultural part of the Unix experience. Like PHP (and Visual Basic on the windows side), it has also helped in allowing people who may not have the aptitude to be real programmers create solutions for the simple problems. All these languages share the same problem, though, one that directly results from the class of programmers they attract and the amount of appearant power they give these programmers to let a project rise above their level of competence.

I've known many a perl programmer with starry eyes delusions of grandeur in their first few years as a developer, something I dubbed the 'pernel' syndrome: A senseless conviction that they could write an OS kernel (or a webserver, or whatever) in perl. I'm not arguing about the possibility as such, but it is fairly typical how a large subset of such programmers get so stuck in their investment in learning the language that they lose perspective on its actual purpose.

Higher level languages are there to make a programmer's life easier. It's a trade, though. A lot of complexity that comes with general purpose programming is taken care of by the language, at the price of losing finer-grained control over the decision process. The same argument goes for all-encompassing APIs and class libraries, by the way. As long as you are precisely aware of the effects on performance each such shortcut has, there is no worry. Most people taking such shortcuts, however, have no idea about how expensive certain operations can be if you do them a couple dozen times per second or operate on datasets that are a couple of scale factors larger than you originally tested the code on.

There is very little that can be done about this, though. The sad reality has always been that there are many more programs to be written than there are expert programmers. Even before the rise of the internet, this was the case. There have been numerous studies that tried to pin things like programmer productivity. Although the latter is hard to measure, such studies uniformly show that the top X% of professional programmers are Y times more productive than the bottom slice, where the value for Y is generally something shocking in the range 10-30. It is also a given that there is more work to be done than can be picked up by this X% and that is where the combination of higher level programming languages and Open Source infrastructure have been and will be of tremendous help in keeping the industry afloat.

Put all this in the context of scoop and you can very well come to the conclusion that, despite the fact that it appearantly stretches perl's design parameters beyond its limits, there would be very little gain in moving it to C/C++ with the current people involved. And getting people who are able to wrangle complex code and optimize for performance is never going to be easy, if only because a piece of web-driven forum software is not that big of a challenge for them to begin with. You can be sure that several of such people are actively contributing to the perl and PHP projects, though, so that is the safest bet.



[ Parent ]
Sort of (none / 1) (#86)
by trhurler on Wed May 25, 2005 at 09:38:26 PM EST

The big problem is that most often, a very small group of people makes 99% of the useful contributions, even in open source systems. Their dedication is such that the barrier to entry really doesn't matter. The biggest reason perl is ideal for this sort of thing is that unless someone is being paid to write it, it just isn't worth the effort to do it in Java or C# or whatever. If there was an online community somewhere that actually needed the performance gains and couldn't more easily obtain them with hardware upgrades, then that'd be a meaningful consideration, but there just isn't.

--
'God dammit, your posts make me hard.' --LilDebbie

[ Parent ]
Yeah (none / 1) (#90)
by rusty on Wed May 25, 2005 at 11:35:13 PM EST

Their dedication is such that the barrier to entry really doesn't matter.

I would modify that slightly, to say that the social barriers to entry matter a lot more than the technical ones. A project that rewards contribution will do better than one that dosn't. Which category Scoop is in I leave to someone else to judge.

The biggest reason perl is ideal for this sort of thing is that unless someone is being paid to write it, it just isn't worth the effort to do it in Java or C# or whatever.

That's true. Perl is fun to write. Very few other languages are.

There's also the fact that Scoop spends vastly more time waiting for mysql than doing anything else. If the processing of an average pageload were the history of the earth, total mysql waiting time would take from the cooling of the mantle until the first homo sapiens appeared, and actual perl processing time would be from then until now.* Writing it in C would probably get us from the time of the first homo sapiens to, maybe, 1945 instead of now.

* Example may not be to scale. But it's closer than assuming time spent running perl code makes a damn bit of difference.

____
Not the real rusty
[ Parent ]

Yes, which is why... (none / 1) (#94)
by skyknight on Thu May 26, 2005 at 07:23:53 AM EST

I vote for intelligent backgrounding of the processes that take forever to run. Also, maybe we could have a VIP server for people who've bought subscriptions, or in my case, people who have finagled them. :-)

It's not much fun at the top. I envy the common people, their hearty meals and Bruce Springsteen and voting. --SIGNOR SPAGHETTI
[ Parent ]
Well (none / 0) (#101)
by trhurler on Thu May 26, 2005 at 09:31:41 PM EST

The biggest part of my now largely abandoned idea to do something better was to do a better job with the database. Excessive string searching is just not the way to go.

--
'God dammit, your posts make me hard.' --LilDebbie

[ Parent ]
so true! (none / 0) (#108)
by dimaq on Fri May 27, 2005 at 08:31:49 AM EST



[ Parent ]
I'm with you on the alarm script thing. (2.50 / 4) (#33)
by Reynolds Number on Tue May 24, 2005 at 09:47:23 PM EST

But only if you can make it play that audio clip from Das Boot where the guy yells "ALARM!" whenever K5 starts flaking out.

Performance problems aren't usually slow servers (2.85 / 7) (#36)
by MichaelCrawford on Tue May 24, 2005 at 10:36:49 PM EST

but slow software. That is, scaling problems, having to wait to grab locks, poorly architected database schemas and things like that.

I was inspired to write a lengthy article on stress testing web servers when I got a frantic phone call in the middle of the night from a client, desperate because his eCommerce application had gone live just in time for the Christmas shopping season, only to find that it fell over under a load of only fifty users, this despite the fact that they had invested hundreds of thousands of dollars in very high-end Solaris servers.

You see, his "developers", if you could call them that, had never stress-tested their code before deploying their application live on the Internet. I suspect they never tested it with more than a few simultaneous users, and probably had only done so by hand, by clicking links in a browser, rather than by running any kind of automated script.

There are actually quite a few load generators available for web applications, as well as for other kinds of network applications. I say in my article that the first load generator I ever saw was for voice mail machines, and would leave hundreds of messages simultanously all day long.

Let's do lunch today.

Now, I expect any Scoop developer would find my article helpful and informative, but if I were to tell you where it was, you'd just complain that I was whoring links. So you'll just have to find it on your own. Maybe google will be of assistance.


--

Live your fucking life. Sue someone on the Internet. Write a fucking music player. Like the great man Michael David Crawford has shown us all: Hard work, a strong will to stalk, and a few fries short of a happy meal goes a long way. -- bride of spidy


Oh that's sneaky... (3.00 / 2) (#74)
by wobblywizard on Wed May 25, 2005 at 01:46:21 PM EST

but you can't bring me to click on your article! This guy here is far to resistant to your subliminal reverse psychological tricks, sir!

No, you're not going to make me say that it's available at http://linuxquality.sunsite.dk/articles/webapptesting/loadgenerators.html!

I won't give you any publicity, nor will I read a single word you wrote, I'm far to crafty to fall for your scheme! Ha, serves you right!

--
You never win an argument with anyone who fucks you or signs your paychecks. I just smile, bite my lip and sip my drink. --Philalawyer
[ Parent ]

Proxy Whore /nt (none / 0) (#77)
by skyknight on Wed May 25, 2005 at 03:31:25 PM EST



It's not much fun at the top. I envy the common people, their hearty meals and Bruce Springsteen and voting. --SIGNOR SPAGHETTI
[ Parent ]
Didn't you get the memo? -nt- (none / 1) (#82)
by wobblywizard on Wed May 25, 2005 at 07:02:03 PM EST


--
You never win an argument with anyone who fucks you or signs your paychecks. I just smile, bite my lip and sip my drink. --Philalawyer
[ Parent ]

Interesting article (none / 0) (#80)
by janra on Wed May 25, 2005 at 05:46:36 PM EST

I wouldn't mind talking to you about how to best stress-test Scoop at some point. Obviously just hitting the front page won't do it, because there are quite a few other, heavier pages, and inserts are slower than selects, and all that.
--
Discuss the art and craft of writing
That's the problem with world domination... Nobody is willing to wait for it anymore, work slowly towards it, drink more and enjoy the ride more.
[ Parent ]
Hey, I have an idea for ya (2.00 / 5) (#38)
by maynard on Tue May 24, 2005 at 11:01:14 PM EST

Sick of paying rusty for service you don't like? Hey, I have an idea! Don't give him money again. *shock!* Problem solved. :) --M

Read The Proxies, a short crime thriller.
What I like about k5. (3.00 / 13) (#39)
by IceTitan on Tue May 24, 2005 at 11:07:43 PM EST

You can say damn near anything and the response will be just as absurd. You could say, "The other day, I was fucking this goat and...". And you'll get a reply along the lines of, "I thought I told you to stop fucking my mother."

How's the saying go? Come for the technology, stay for the trolls?
Nuke 'em from orbit. It's the only way to be sure.

Are you saying... (none / 1) (#104)
by SoupIsGoodFood on Fri May 27, 2005 at 02:20:14 AM EST

That rusty should be fixing rofflecopters instead of K5?

[ Parent ]
I pay for K5 ads by the impression, not the click (2.71 / 7) (#40)
by MichaelCrawford on Tue May 24, 2005 at 11:57:20 PM EST

I should clear up a minor misunderstanding that skyknight has about k5's text ads. It's not actually a real problem that one can view advertiser URLs by hovering one's mouse over the link in a text ad. If one then goes directly to the page, the worst that happens is that K5 is not credited for all the traffic that the advertiser gets, but K5 doesn't lose money.

The reason is that K5 charges by the impression rather than by the click. The ads you see on the front page are called Text Ads or Expanded Text Ads. These are ten bucks for ten thousand impressions. The tall, more verbose ads in story bodies are Story Text Ads. These are ten bucks for twenty thousand impressions.

Both kinds of advertising are widely used, depending on the site. I think one reason that pay per click is probably more common is that it is more difficult to be sure your ad is actually getting served for the number of impressions you pay for. On the other hand, unscrupulous ad publishers can click the ads on their own site to steal money from advertisers, a problem that one of Google's founders called "A serious threat to our business model".

What matters in each case is how much you pay per click. After letting a ten dollar K5 ad placement expire, divide ten dollars by the number of clicks reported. The best I've been able to achieve at K5 is ten cents a click, with some of the Text Ads I've placed. These are the ones that allow less text (35 characters in the title I think) but have larger fonts. The worst I've seen were the Story Text Ads, which cost me forty to eighty cents a click. I stopped running them very early in my testing.

Google likes pay per click because Google AdWords Select are based on keywords, either from search terms entered into the Google search engine, or predominant keywords in the content pages of Google AdSense publishers. AdWords select publishers bid to get the best placement on the pages, or even to appear at all.

Google gets paid as much as twenty dollars a click for some highly competitive keywords, for example having to do with asbestos lawsuits. On the other hand, one can place ads for less-competitive keywords, that might still get a lot of clicks, for as little as five cents a click - even better than K5.

The key to pay per click advertising is to either pay very little, to collect a lot of clicks, or to be very sure that anyone who will ever be likely to click your ad will be definitely wanting to buy what you've got to sell - for example, people who have been injured by asbestos and are looking for a personal injury attorney.

There are a couple other ways one can click k5 ads without the clicks being counted. One is to click the "active" link below a text ad to view K5's active ads. Clicks on that page aren't counted. One can also click the "comments" link on an individual ad to bring up the story associated with that ad. Here's one of mine.

The worst thing about this isn't that K5 is losing money directly, but that the advertiser is getting a little traffic that K5 isn't claiming credit for. If the difference were significant, the advertiser might decide that a click at k5 costs too much money, and not place any more ads, but I don't think that so many people click the kinds of ads that don't track clicks.

One thing I am very surprised at is that Google allows AdWords select publishers to display a URL distinctively and directly as the bottom line of their ad. What this means is that someone might remember the URL and just go to it by entering it manually in their browser. Google doesn't earn anything when that happens, or even get any credit.

Sometimes when I see an ad for something I feel strongly positive about, I might enter the URL manually to go see the advertiser's site without costing them any money.


--

Live your fucking life. Sue someone on the Internet. Write a fucking music player. Like the great man Michael David Crawford has shown us all: Hard work, a strong will to stalk, and a few fries short of a happy meal goes a long way. -- bride of spidy


Conversely (3.00 / 3) (#44)
by Eight Star on Wed May 25, 2005 at 02:00:04 AM EST

I click on ads of causes I oppose to cost them money.

[ Parent ]
Saving them money... but at what cost? (none / 1) (#72)
by Chancellor Martok on Wed May 25, 2005 at 01:06:45 PM EST

But if you visit their site manually, they won't have the statistic to show that you found their site through the advertising. As this sort of behaviour accumulates, they may think the advertising isn't working and pull it, instead moving to a different advertising location/scheme that might actually be worse performing.

-----
Chancellor Martok  in Sydney, Australia
"Castrate instead. That can surely rehabilitate. I did it volunatrily, and my grades went up!"  -- Sen

[ Parent ]
But who are you... (none / 0) (#73)
by skyknight on Wed May 25, 2005 at 01:21:41 PM EST

to decide that one person's resources are more important than another's? What right have you to mess with the contract made between advertiser and advertisee? Were you an advertiser, would you not find such vigilante moralization irksome? It's easy to conjure up in your head the vision of some mega-corporation with ultra-rich board members being deprived of a couple of dollars so as to save that money for a desperate and impoverished cause, but how often is this really the case? How do you know which case it is? How do you know that the advertiser is not a small business owner with a mortgage to make and kids to feed, and a bank account whose ledger only reads four digits? Do you feel that the advertiser is not entitled to compensation for the services they provided, for which they undoubtedly have expenses? Wouldn't the far more noble thing to do be to send a check to the advertisee if you think that their cause is worthwhile?

It's not much fun at the top. I envy the common people, their hearty meals and Bruce Springsteen and voting. --SIGNOR SPAGHETTI
[ Parent ]
Four digit bank accounts!? (none / 1) (#78)
by rusty on Wed May 25, 2005 at 05:08:16 PM EST

Rich bastards. We should eat them.

____
Not the real rusty
[ Parent ]
I mock you (2.42 / 7) (#42)
by trhurler on Wed May 25, 2005 at 12:47:58 AM EST

Regarding the monitoring idea: it would work, but to really do it right(ie, to really know if there is a problem,) is a bit tougher than just checking the front page, and besides, users bitch pretty quickly anyway.

Regarding the decline of the Holy Roman Empire: yeah, that happens.

Regarding that pony: are you sure you wouldn't rather have a ponie?

Regarding your angst over all of this: see the subject line.

In conclusion, HAHAHAHA. Unable to connect to SSL server. Hehe... I connected to your mom's port and sent her some bits with NO PROBLEMS.

Oh, and I'm kind of tipsy too. Heh.

--
'God dammit, your posts make me hard.' --LilDebbie

Well, yes... (2.66 / 3) (#66)
by skyknight on Wed May 25, 2005 at 11:24:56 AM EST

Regarding monitoring, you'd want to test a variety of things. Users may bitch, but users are unreliable. You don't know the actual parameters of their woes. For instance, I e-mailed help@kuro5hin.org once because I forgot that my iptables configuration script took a special argument that specified whether or not to block all traffic to K5. Aphrael was highly amused. :-)

Regarding the Holy Roman Empire, yes there is nothing more cliched than bemoaning the inevitable fall of K5. I'm sure it was very tedious for the Romans to hear about the inevitable fall of Rome for hundreds of years before it transpired, and I'm sure it is similarly tedious for Rusty, but Rusty doesn't have an army and he's not God on Earth.

Regarding ponies, I have no actual interest. I was engaging in what is known as literary flourish. I would, however, be OK with a million dollars. Now that I'm done with graduate school I have actual time to start thinking about how to amass said wealth. I have ideas.

Regarding my purported angst: your mother!

Regarding my mother, she would really appreciate it if you'd stop referring to your hand thusly.

Regarding your tipsiness, that would put you in for what? Two Zimas? You should gain some weight to deal with that.



It's not much fun at the top. I envy the common people, their hearty meals and Bruce Springsteen and voting. --SIGNOR SPAGHETTI
[ Parent ]
A drinking contest? (none / 0) (#85)
by trhurler on Wed May 25, 2005 at 09:24:19 PM EST

Hey, I might weigh 50-100 pounds less than you, but I will drink your ass under the table.

--
'God dammit, your posts make me hard.' --LilDebbie

[ Parent ]
So you claim... (none / 0) (#88)
by skyknight on Wed May 25, 2005 at 10:54:21 PM EST

baselessly. I thought you fashioned yourself a man of science.

It's not much fun at the top. I envy the common people, their hearty meals and Bruce Springsteen and voting. --SIGNOR SPAGHETTI
[ Parent ]
What does science have to do with anything? (none / 0) (#100)
by trhurler on Thu May 26, 2005 at 09:30:17 PM EST

I have a tolerance you can't match:)

--
'God dammit, your posts make me hard.' --LilDebbie

[ Parent ]
This may be late, but... (none / 0) (#136)
by beergut on Wed Jun 08, 2005 at 09:08:06 PM EST

I think you did not understand what he said.

He was actually propositioning you.

i don't see any nanorobots or jet engines or laser holography or orbiting death satellites.
i just see some orangutan throwing code-feces at a computer screen.

-- indubitable
[ Parent ]

I don't get it (none / 1) (#48)
by LodeRunner on Wed May 25, 2005 at 04:30:42 AM EST

Looking at the queue score, people seem to be agreeing with you (57 at the time of this writing).

But in the poll, "Acceptable load times" is winning by far (79%), contradicting your main point.

What gives??

---
"dude, you can't even spell your own name" -- Lode Runner

You forgot - BEST ENDING EVAH (nt) (none / 0) (#51)
by tetsuwan on Wed May 25, 2005 at 08:42:31 AM EST


Njal's Saga: Just like Romeo & Juliet without the romance
[ Parent ]

I know! What the hell? (none / 0) (#54)
by skyknight on Wed May 25, 2005 at 09:51:34 AM EST

This just goes to show that sampling bias renders pretty much all statistical calculations useless. Presumably the people who are bitter about K5 being slow are somehow predisposed to eschewing poll voting.

It's not much fun at the top. I envy the common people, their hearty meals and Bruce Springsteen and voting. --SIGNOR SPAGHETTI
[ Parent ]
Most loading can be done in background tabs (none / 0) (#68)
by Phssthpok on Wed May 25, 2005 at 11:49:31 AM EST

but submitting a form consumes the active tab. I don't have all day to wait for my vote to be counted! I want to read comments while the story is fresh in my mind.
____________

affective flattening has caused me to kill 11,357 people

[ Parent ]
The credit card thing (none / 1) (#53)
by rusty on Wed May 25, 2005 at 09:46:44 AM EST

The processor changed the API out from under me, it appears. Lovely. So I have to recode to use the new version. Fortunately it appears that CPAN has some modules standardizing billing stuff now, so at least I can probably make the whole thing more consistent.

The page load is almost certainly due to the new comments watcher. Ironically, without a subscription you'll probably have faster load times. If you noticed that it'll be slow for the first page, then fine for subsequent pages after that until you take a break for a while, then that's what it was. Even with that, I haven't seen big two-minute lags that time out. Has anyone else?

____
Not the real rusty

Well, in fact... (none / 0) (#59)
by skyknight on Wed May 25, 2005 at 10:17:23 AM EST

we have had this conversation previously. I took that into account. I think what really irked me was that on Sunday I wasn't even a subscriber anymore, and I was experiencing really long delays and even time outs. It didn't exactly put me in the mood to renew my subscription, as much as I love K5, and as much as I really do want to re-up my membership. Also worth noting, it's extrememly unlikely that this is a network issue on my part. I've had similar experiences at my apartment, at my parents' house in another locality, and on my school's campus. I figured that it must have been something other than comment reply notification generation since I wasn't a member anymore, but maybe there's a bug such that if you have your membership expire the generation is still occuring but you just don't see the results?

Also, here's an architectural thought about comment reply notification... When I start surfing K5, do I really care if I see all of my latest comment replies? Or would it be OK if I saw them trickle into my list over the next few clicks whilst being favored with blazing fast page load times? I think what would be ideal would be if instead of making a page load contingent upon reply notification freshness, said page load kicked off a background process that takes care of it, placing the results somewhere that the viewer process can retrieve them when they are finished. It's just really miserable to have to wait around for an extended period of time while your reply cache is dragged kicking and screaming up to date. I'd be perfectly happy to peruse the latest stories and diaries while I waited for my comment reply notifications to be calculated.

Here's another idea... How much idle CPU time do K5's servers have in the course of a day? Obviously there are the times of the day where K5 is utterly swamped, but signing on late at night, I've noticed that there are times when K5 is basically unused, but for a dozen people. Why not use these times productively, much like power plants might pump water into an elevated resevoir to store surplus generated power? Consider this... Instead of having K5 update users' reply notification caches the first time they load a page after a while, why not have some kind of work queue system in which there is a background process soaking up idle CPU whenever it becomes available to generate this stuff, placing it somewhere that the viewer module knows where to find it? You could normally run this as a process with the maximum "nice" value so that it would only run when there was nothing else to do. Of course, you'd have to have something on top of this, to deal with starvation. You don't want a heavy load to effectively disable comment reply notification... I'd have to think about this. It's a very interesting problem.

Lastly, how long until the order form is fixed? Not having reply notification is an unbearable nuisance, and I don't really want to go to the trouble of writing my own screen scraper and caching code, not if I'm ultimately going to continue paying for membership. Wanna cut me some slack and reactivate my membership until the CC server works again?



It's not much fun at the top. I envy the common people, their hearty meals and Bruce Springsteen and voting. --SIGNOR SPAGHETTI
[ Parent ]
You're right (none / 0) (#61)
by rusty on Wed May 25, 2005 at 10:32:39 AM EST

First, no there's no way it's slowing you down if the comment replies aren't showing. Right now those are being processed by the box that shows them. If the box isn't there, the code doesn't run. So that can't be it.

But you're right that it needs to be doing this stuff in the background. What would seem to make sense would be if there was a process collecting reply notifications for display that was triggered by a flag that gets set when someone replies to a comment. So, for example, I reply here, and it sets your flag, which tells the watcher to pull up your replies when it gets to that flag in the list. Or some such thing like that.

As for when the CC will be fixed, as soon as possible. I hooked you up with a couple months, because you obviously value it, and it isn't your fault you can't pay me. :-)

____
Not the real rusty
[ Parent ]

Sweet. Thanks. (none / 1) (#65)
by skyknight on Wed May 25, 2005 at 10:59:11 AM EST

Except... It doesn't seem to have gone active. I have neither reply notification nor spell check. Maybe some hook or other did not get invoked because you did something manually?

I guess the big concern with having a background process or processes churning out reply notification autonomously would be swamping the disk. Maybe as a safe guard there could be a flush daemon of sorts, much the same way that virtual memory works on most operating systems. What is the main rationale for the present system, and what are the main perceived drawbacks about alternate systems? Why not just add an item to a reply queue for a user whenever someone replies to his comment. For that matter, would such a system scale nicely into being able to offer reply notification for all ancestral comments? That would be very cool. I can't tell you how many times I've gone back to an old comment on a whim and found that a really neat thread developed below it. I wonder how often I miss out on cool stuff as a result of lacking such a feature.



It's not much fun at the top. I envy the common people, their hearty meals and Bruce Springsteen and voting. --SIGNOR SPAGHETTI
[ Parent ]
Oops (none / 0) (#69)
by rusty on Wed May 25, 2005 at 12:05:20 PM EST

I forgot a step. It should be active now.

About notification -- the way it works now is nice because it just derives the notice from information that's already being tracked. We didn't have to add any special tables or fields -- it just implies the new comments from tracking what you've viewed. The drawback is it doesn't do any kind of ongoing thread tracking -- it only looks for direct replies. Doing it the way I described above could easily scale to include full-thread watching, just by setting a flag for each person in the thread above a new comment.

I don't think swamping anything would be a big concern. On further thought, there'd be an even simpler way to do all this. Have a "replies" table that stores uid, sid, cid (of your comment) and cid of the reply. When someone posts a new comment, it fires a hook that walks up the thread from their comment and adds a record for each of its ancestors with the ancestor's uid, the story's sid, and the new comment and ancestor comments cids.

When you load a page, the replies box grabs any records marked with your uid from the replies table. It checks viewed_stories to see whether you've already looked at any of them, and removes the records of any you have. Then it formats the rest up and displays them.

This would be a lot faster, even if the replies table got big, because all the queries are either plain inserts or selects on an indexed column of small values (uid). The slowness in the current version comes from the fact that it joins comments, comments (again), and viewed_stories which are probably the two biggest and hairiest tables in the whole thing. It's an ugly join. The above idea would probably be a much faster way to go.

____
Not the real rusty
[ Parent ]

One more step? (none / 0) (#75)
by janra on Wed May 25, 2005 at 01:53:50 PM EST

When you view a story (or even the comment) it is removed from the replies table, as it is no longer a new reply.

That would stop all those complaints about having to load the entire story to clear the 'new replies' box. (It wouldn't fix the [new] marker or "1 new comment" when you've just posted, but clearing new replies seems to confuse a lot more people.)
--
Discuss the art and craft of writing
That's the problem with world domination... Nobody is willing to wait for it anymore, work slowly towards it, drink more and enjoy the ride more.
[ Parent ]

Yeah (none / 0) (#76)
by rusty on Wed May 25, 2005 at 02:11:54 PM EST

That's another good idea. Any page that loads a full story worth of comments would remove any comments below the current highest cid for that story, and any page that shows a comment at all would remove that particular one from your replies table, if it's there.

____
Not the real rusty
[ Parent ]
linkpoint problem? (none / 0) (#71)
by urdine on Wed May 25, 2005 at 12:29:17 PM EST

If rusty is using Linkpoint, they've just moved their servers and are having a lot of intermittent problems.  Maybe unrelated completely, but the world of e-commerce is a mountain built on toothpicks.

You got it (none / 1) (#79)
by rusty on Wed May 25, 2005 at 05:09:50 PM EST

It's probably my problem. I have to upgrade to their new API. I finally dug up the email from several months ago (which referenced other communication they never had with me) about it. All in all, this was not a well-managed upgrade.

____
Not the real rusty
[ Parent ]
You might think... (none / 0) (#96)
by skyknight on Thu May 26, 2005 at 07:33:57 AM EST

that they would leave what you were using as a deprecated API and heckle you if you continued using it. Of course, then you'd be guilty of harboring unreasonable expectations of reasonableness.

It's not much fun at the top. I envy the common people, their hearty meals and Bruce Springsteen and voting. --SIGNOR SPAGHETTI
[ Parent ]
Linkpoint Update (none / 0) (#123)
by rusty on Sat May 28, 2005 at 04:52:18 PM EST

Linkpoint are a bunch of rancid goatfuckers.

I got the new API, and discovered that they changed it just enough so that none of my old code works, but not quite enough to hide the fact that they could have made it totally backward-compatible with the old API if they'd felt like it. In fact, what I did was mainly just find the one or two places where all my processing code talked to their API and rewrote a bunch of field names there to match their new input field names.

So now all I can get back from their gateway is "invalid XML", which is entirely not true, since there's nothing wrong with the XML that their own fucking library is producing.

Meanwhile their status page still reports intermittent errors of unspecified type. I have a support request in, since I get the same unhelpful errors from two different sites using their piece of shit gateway.

All I can say is, friends don't let friends sign up with CardService.

____
Not the real rusty
[ Parent ]

If I were the provider of such a service... (none / 0) (#126)
by skyknight on Sun May 29, 2005 at 08:03:34 AM EST

and I felt compelled to change my API, and furthermore to eventually eliminate the old API, I think what I would do is leave the deprecated API in place, but whoever used it would get an e-mail once per day warning them that they were using a deprecated API. If after several months of ignoring e-mails, then I might try a telephone call to each of the people still using the deprecated API, and some duration after that go ahead and break it. The only reason I can see for immediately breaking an API is if there is a security hazard of some sort that has been discovered. Short of that, it's just lunacy to fuck over paying customers in such a cavalier fashion.

On another note, is there anything that you could do to detect these problems sooner? Could you reasonably test the correctness of external APIs in real time in an automated fashion? It would be nice if these providers had a dummy interface where customers could submit fake payments so as to verify the correct functionality of their code. Short of that, maybe you could make one cent payments periodically. I don't know whether that's worth the trouble, though.



It's not much fun at the top. I envy the common people, their hearty meals and Bruce Springsteen and voting. --SIGNOR SPAGHETTI
[ Parent ]
They claim they did (none / 0) (#128)
by rusty on Sun May 29, 2005 at 01:46:02 PM EST

I got one email about a month ago referring to other communication I never had with them about changing APIs. And supposedly the old one doesn't cut off until June 15th, but it has already stopped working, so who the hell knows.

I've read all the documentation and help I can find about this change, and every damn thing has different information. Different version numbers, different dates, etc etc. My impression is this company couldn't find their own ass with two hands and a flashlight.

They have all kinds of testing crap, but my experience has been that none of that ever works. The first time I went through deploying all of this I did it with the test servers first, got it all working, then switched it live and nothing worked. I've found it much easier to just spend a couple bucks testing. Which I'd have no problem with, if the live code worked.

My support email was form-lettered back today with no helpful information. I will apparently have to keep trying.

____
Not the real rusty
[ Parent ]

Well, this just goes to show... (none / 0) (#129)
by skyknight on Sun May 29, 2005 at 03:26:00 PM EST

that software development and operation is a problem that goes far beyond technical issues. The problems that you are experiencing are not so much technical failures as social ones. They clearly don't understand how to behave decently, something that is a prerequisite for creating software systems that are pleasant (or even tolerable) to use.

How hard would it be for you to tell them to FOAD and sign up with someone else? Also, is your code architected in a pluggable fashion, such that you have a high level API with which Scoop interacts, and a facility for dynamically plugging in arbitrary implementations for actually processing the billing information?



It's not much fun at the top. I envy the common people, their hearty meals and Bruce Springsteen and voting. --SIGNOR SPAGHETTI
[ Parent ]
Yes and no (none / 0) (#130)
by rusty on Sun May 29, 2005 at 04:19:29 PM EST

At this point, the hassle of me switching to some other processor would exceed the hassle of waiting till they figure out what's wrong with their stuff. There's a lot of preliminary crap to deal with to open a merchant account -- paperwork, corporate documents, etc. I can put up with not having CC processing for a week or two while this gets sorted out easier than I could do all that over again. I won't be signing anyone else up with them in the future though.

And yeah, this is mainly social. Although it is partly technical -- their code doesn't work. :-) Financial code is generally nightmare though. Legacy financial systems are already ugly, and they have no real incentive to make their interfaces anything but minimally functional, since the decision to use one processor or another is typically made by people who won't ever have to even think about whether the code's any good, and the work of implementing the code is usually done by people who have no choice in the matter.

As for high-level interfaces, sort of, but not really. Scoop has a Billing.pm module that handles the things that Scoop itself currently does billing for (ad buys, ad renewals, and subscriptions). Those methods use other methods from Billing/Linkpoint.pm and Billing/Paypal.pl, which have general methods that know how to talk to those two financial systems. But it's not very generalized -- it's not like all the method calls are the same. Billing.pm still has to know what kind of transaction its doing and call methods appropriately.

What I need to do is use the CPAN Business::OnlinePayment modules to make it so that we can use any CC processor without changing any Scoop code. But at this point, my goal is to make linkpoint work again, so that's going to have to wait a little bit.

____
Not the real rusty
[ Parent ]

Let me known what's going on. (none / 0) (#131)
by Inoshiro on Sun May 29, 2005 at 09:26:35 PM EST

I can always investigate alternatives and also work around on these problems. I'm "down to" 45-ish hours a week of work, so I have an evening or so a week I could use to look into stuff if we known a breakage is coming.

--
[ イノシロ ]
[ Parent ]
Hey! (none / 0) (#132)
by rusty on Sun May 29, 2005 at 11:18:58 PM EST

What have you been up to? :-)

____
Not the real rusty
[ Parent ]
Ahah! (none / 0) (#134)
by Inoshiro on Mon May 30, 2005 at 02:00:44 AM EST

Someone else who doesn't read my userinfo page ;)

I've been up to working on my BSc Honours.  It's going to take a while since I have to do it around jobs.  Mainly that since full time school + ~24 hours a week of work = busy me.

Once I'm done my BSc I'm thinking about MSc or PhD; PhD would mean I'd have to relocate to another uni if I wanted to be a prof (which I have as a long-long term goal).

In the meantime, I'm doing a co-op pos at PMC-Sierra doing Linux kernel work.  It's very rewarding.

--
[ イノシロ ]
[ Parent ]

Paying for things one values is a good thing! (none / 0) (#84)
by Oldest European on Wed May 25, 2005 at 08:25:13 PM EST

The real reason [...] is that I like to pay for the things that I enjoy.

And what better reason could there possibly be?

And since you value K5 you can even say that you make your world a better world by sponsoring it.

Sure K5 might still survive without your bucks, just as some open source project might survive without you submitting patches, but if nobody would contribute, sites like K5 couldn't exist or would be sex-and-online-casino-pop-up-ad-nightmares.

I just registered my K5 account this week and have been reading K5 every now and then over last 6 months, but K5 is actually among the few websites where I feel okay with clicking ads.

Don't know yet if I'm going to be a subscriber...

Your voice your money (none / 0) (#137)
by VitaminJunky on Sun Jun 12, 2005 at 12:58:54 PM EST

It sucks when something you once enjoyed has changed and you want the "old" way back. Really the #1 way to make your voice heard is using your money.

Well, yes... (none / 0) (#138)
by skyknight on Mon Jun 13, 2005 at 07:49:23 PM EST

or cyber-terrorism. Unfortunately I'm too boring and square to be a terrorist. I'll just have to make do with writing petulant articles.

It's not much fun at the top. I envy the common people, their hearty meals and Bruce Springsteen and voting. --SIGNOR SPAGHETTI
[ Parent ]
Reaching My Limits | 137 comments (129 topical, 8 editorial, 0 hidden)
Display: Sort:

kuro5hin.org

[XML]
All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!