Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

[P]
Scoop In a Production Environment

By sallgeud in News
Tue Jun 06, 2000 at 03:01:45 PM EST
Tags: Scoop (all tags)
Scoop

Three months ago we were approached about an application we had discussed in a previous meeting with a different department in our bank. At the time, they had just split off to form their own company. The bank's CEO left to be their leader. In place, at the time, was a discussion application created by Organic Inc. IMHO, it was one of the most horribly written piles of java I've ever seen. The sad part of it all is that we had told the business development group that our developers would have no problem creating a discussion based news page. In the end they shelled out in the upper 6 figures for what ended up being trashed only 6 months later. Which now leads us to scoop...


So, being approached about this discussion server we had discussed, I mentioned a free set of software that we were considering using (slash). After talking with a few of the people over at eScout (the aformentioned 6 figure spending company), I decided I should call a meeting with some of the members of our group to discuss what we would be willing to do for them. In the meeting, Al, one of the technical leaders in our group, mentioned another news based discussion app (Scoop). I remember it being said that, "It's probably not possible that it could be more bloaty than slash, so maybe we should take a look.".

The next day, I get a link sent to myself from Al. I check out the site, download the source, install it on my personal FreeBSD 3.4/P 133/32M. Of course, this required the removal of all that bloaty slash code. It took me about 20 minutes in all to install scoop and have it running properly. My first impression (without looking at code)... amazement. I hadn't seen an app perform this well in ages. I sent out an email to Rusty informing him of our interest in being involved with this project.

It's now 4 weeks later, and scoop is up and running. Everyone seems to love it. The minor changes we made have been commited to the repository; and for the first time in a long time, I'm truely enjoying my job.

Anyone who'd like to check it out... Insight.eScout.com

A note on some minor changes in our app that differentiates it from the scoop you would download...

  • Different security levels for different features
  • Inability for Anonymous users to post responses
  • Changed some column names to enable it to work with Oracle in the future (hopefully commited when bugs are worked out)

Sponsors

Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure

Login

Related Links
o Scoop
o different department
o our bank
o they
o their leader
o Organic Inc
o eScout
o them
o Rusty
o up and running
o Insight.eS cout.com
o Also by sallgeud


Display: Sort:
Scoop In a Production Environment | 30 comments (30 topical, editorial, 0 hidden)
Hey, cool. It's nice to hear of ou... (none / 0) (#4)
by fluffy grue on Tue Jun 06, 2000 at 12:45:37 PM EST

fluffy grue voted 1 on this story.

Hey, cool. It's nice to hear of our favorite database-driven discussion site being used in a "real" environment.
--
"Is not a quine" is not a quine.
I have a master's degree in science!

[ Hug Your Trikuare ]

Re: Hey, cool. It's nice to hear of ou... (none / 0) (#13)
by rusty on Tue Jun 06, 2000 at 03:06:45 PM EST

It's nice to hear of our favorite database-driven discussion site being used in a "real" environment.

And what kind of environment is this, then? Chopped liver? ;-)

____
Not the real rusty
[ Parent ]

Re: Hey, cool. It's nice to hear of ou... (none / 0) (#29)
by fluffy grue on Wed Jun 07, 2000 at 01:14:38 PM EST

It's just not a "real" (as in suit-and-tie ecommerce media-hype) environment, is all. :)
--
"Is not a quine" is not a quine.
I have a master's degree in science!

[ Hug Your Trikuare ]
[ Parent ]

Re: Hey, cool. It's nice to hear of ou... (none / 0) (#30)
by rusty on Wed Jun 07, 2000 at 01:49:31 PM EST

That's true. It is a "real" (as in fanatical nerds, reloading constantly and inputting data that breaks things) environment, though. :-)

____
Not the real rusty
[ Parent ]
What's to discuss? Scoop rocks :-)... (none / 0) (#9)
by slycer on Tue Jun 06, 2000 at 12:50:28 PM EST

slycer voted 0 on this story.

What's to discuss? Scoop rocks :-)

Intresting, and good to see! any ch... (none / 0) (#8)
by SgtPepper on Tue Jun 06, 2000 at 12:53:20 PM EST

SgtPepper voted 1 on this story.

Intresting, and good to see! any change of posting diffs to the code you changed? ( the security levels for the features intriques me )

Re: Intresting, and good to see! any ch... (none / 0) (#19)
by Alhazred on Tue Jun 06, 2000 at 05:11:47 PM EST

I can give you a set of classes that do user/group type security for CGI's using a MySQL database. I'm sure they can be integrated into scoop easily. In fact I'm on the path of doing that now, just had to set up a new Linux box to run it on (mod_perl poops on my SGI for some reason, oh well).

To be honest I'd like to see the functionality of Scoop seperated out more into a set of application classes and interface classes. I really prefer for instance to use fastCGI, as it scales better than mod_perl, but as it is now I'd have to basically rewrite scoop to do that.
That is not dead which may eternal lie And with strange aeons death itself may die.
[ Parent ]
Re: Intresting, and good to see! any ch... (none / 0) (#21)
by rusty on Tue Jun 06, 2000 at 05:52:52 PM EST

The user/group classes would probably solve a lot of problems. Scoop kind of relies on having info about the users locally, or at least locally available, but there's no reason our user tables have to be the source of login authentication. I've wanted to abstract that stuff away for a while, to enable people to use whatever auth mechanism works for them (LDAP, basic auth, whatever)

I agree that it would be better handled as a set of classes or services. However, there is a point at which something can become *too* abstracted, as in impossible to configure and manage, let alone code for or extend. Scoop, as it stands, though, is merely a first effort. There's a lot to be improved, and I wholeheartedly welcome any hacking you'd like to take on. :-)

Oh yeah, the fastCGI stuff-- how much would have to be changed, really? Scoop went from originally being an Apache::Registry-interpreted script, to being a whole-hog mod_perl application with just a tiny tweak. Most of the actual functionality is in libraries that don't much care what their interface to the web server is. The web server *only* ever talks to Scoop::ApacheHandler. I wonder how much trouble it would be to re-write ApacheHandler as a fastCGI interface, and allow users to choose what they want to run... (Note: I've never used fastCGI, so I don't know how it works, or if this even makes sense)

____
Not the real rusty
[ Parent ]

Re: Intresting, and good to see! any ch... (none / 0) (#28)
by Alhazred on Wed Jun 07, 2000 at 10:52:58 AM EST

It makes perfectly good sense ;o)

To be honest, while I have got a play setup going now, I haven't had a chance to do a real close look at the code.

GENERALLY what I've found WRT web based applications is that they lack sufficiently generalized session management, authentication/authorization, and request/response management to be easily modified. Unfortunately most of that is in the nature of the beast.

For the work I've been doing the last couple years I built classes to abstract that stuff into what amounts to a publishing system. Unfortunately you are correct about complexity and manageability. Its not a dirt simple system to use!

As for fastCGI, its really pretty simple. Your application runs as a stand alone long-lived process, and the web server simply uses mod_fastcgi to send it requests via a Unix domain or TCP socket. Fast::CGI is one perl module you can use to work with it. Basically it is identical to the standard CGI.pm module. Each request looks exactly like a CGI invocation to your code, except you sit in a loop looking for requests, like:

while(my $req = new Fast::CGI()) { handle_request($req); }

You can take an existing CGI.pm based app and just stick basically those lines at the top and encapsulate the rest of the script logic in handle_request() and have a fastCGI! Of course like with mod_perl your script lives beyond one request and so you have variables that can retain state etc. The tricky part is that of course a single fastCGI process is basically a daemon that can handle only one request at a time...

You can have many load balanced copies of the same process (mod_fastcgi handles the dispatching) but then you have the "session affinity" problem if you maintain state in variables (can't guarantee the same process handles all of user X's requests). Some tiresome fiddling with mod_rewrite can fix that, but its ugly!

My answer was to use perl 5.6 and use multi-threading. So the "handle_request()" function spawns a thread which does the request handling, while the main loop thread goes back to waiting for a new request, which seems to work awesome, but of course now you have to be thread safe!

Anyway, I'll take a good look at the scoop source this week and maybe I can throw in a patch or two that will make it possible to run in fastCGI, though I suspect it will basically mean abstracting a few things.
That is not dead which may eternal lie And with strange aeons death itself may die.
[ Parent ]

Do we have enough links to escout.c... (none / 0) (#2)
by Pelorat on Tue Jun 06, 2000 at 12:55:37 PM EST

Pelorat voted 1 on this story.

Do we have enough links to escout.com? I mean, we might run out, better toss a few more in there...

=)

Never enough links. (5.00 / 1) (#12)
by sallgeud on Tue Jun 06, 2000 at 03:06:42 PM EST

I should note, I don't work for eScout... I just thought it would be fun to link to different places like a maniac, whenever it happened to be relevant. It was intended as humor... though probably ended up being one of the best advertisements they've ever had.

[ Parent ]
Re: Never enough links. (none / 0) (#15)
by SgtPepper on Tue Jun 06, 2000 at 03:14:58 PM EST

So would that be NSMLP? (NS = Not So )? :)

[ Parent ]
Speaking of Never enough links. (none / 0) (#18)
by Pelorat on Tue Jun 06, 2000 at 04:04:40 PM EST

There's a Windows 2000 news/portal site, can't remember what it is... what it looks like they do is run their pages through a filter, which wraps microsoft.com links around *every* instance of "Microsoft", even if it's mentioned twice in the same sentence. Makes for absolutely hilarious reading...

[ Parent ]
I'm the web adminstrator for the ba... (5.00 / 1) (#7)
by dvicci on Tue Jun 06, 2000 at 01:06:18 PM EST

dvicci voted 1 on this story.

I'm the web adminstrator for the bank sallgued and I work for, which means I'm the web administrator for the aforementioned company as well. (the server performs just fine... the code it's delivering... well, that's another story)

I'm with sallgued, this has been quite the amazing thing to see happen... I never thought I'd see the day I was involved in an open-source project that saw the light of production servers here at the bank.

I don't kid myself into thinking this is one of the first such instances, but it IS the first such instance for our bank (aside from the staples: sendmail, basic protocols, etc.), and as such, has made working here far more enjoyable.

We deployed Scoop at a MINUTE fraction of the cost of their previous discussion server, in a fraction of the time, and it runs orders of magnitude faster, cleaner, and is SO MUCH MORE FUN to maintain. :)



Shouldn't this be posted at http://... (none / 0) (#10)
by Anonymous Zero on Tue Jun 06, 2000 at 01:18:33 PM EST

Anonymous Zero voted 0 on this story.

Shouldn't this be posted at http://scoop.kuro5hin.org/ instead?

Testimonials, etc about Scoop belon... (1.00 / 1) (#6)
by inspire on Tue Jun 06, 2000 at 01:30:24 PM EST

inspire voted -1 on this story.

Testimonials, etc about Scoop belong at: scoop.kuro5hin.org.
--
What is the helix?

Re: Testimonials, etc about Scoop belon... (5.00 / 1) (#11)
by rusty on Tue Jun 06, 2000 at 03:05:31 PM EST

Testimonials, etc about Scoop belong at: scoop.kuro5hin.org.

Which is currently down (I suck). :-)

____
Not the real rusty
[ Parent ]

Re: Testimonials, etc about Scoop belon... (none / 0) (#14)
by SgtPepper on Tue Jun 06, 2000 at 03:12:38 PM EST

Damn straight! What for you posting boy, get back to work and fix scoop! *cracks whip* ;)

[ Parent ]
Please sir may I have another! (none / 0) (#23)
by rusty on Tue Jun 06, 2000 at 05:57:49 PM EST

:-)

____
Not the real rusty
[ Parent ]
Re: Please sir may I have another! (none / 0) (#24)
by SgtPepper on Tue Jun 06, 2000 at 06:46:49 PM EST

Not if you're enjoying it that much *LOL*

[ Parent ]
For the record, sallgeud has been v... (3.00 / 1) (#1)
by rusty on Tue Jun 06, 2000 at 01:46:47 PM EST

rusty voted 1 on this story.

For the record, sallgeud has been very much in touch with me through this whole process, and of course the changes will be released under the GPL along with the rest. Free software in action, baby. :-)

____
Not the real rusty

always nice to give Rusty a well-de... (none / 0) (#5)
by analog on Tue Jun 06, 2000 at 02:04:27 PM EST

analog voted 1 on this story.

always nice to give Rusty a well-deserved pat on the back.

While it is a little bit of adverti... (3.00 / 1) (#3)
by skim123 on Tue Jun 06, 2000 at 02:20:30 PM EST

skim123 voted 1 on this story.

While it is a little bit of advertising, Rusty deserves the ego-stroke this story will no doubt produce! :-)

Money is in some respects like fire; it is a very excellent servant but a terrible master.
PT Barnum


win-win and a suggestion (none / 0) (#16)
by SgtPepper on Tue Jun 06, 2000 at 03:18:59 PM EST

Very nifty, not only is it a win for Scoop, it's a win for Free ( as in speech AND beer ) software.

Now if only Rusty can get CNN to use scoop.....

oh and since scoop.kuro5hin is down...

Any chance of perhaps making a link on the article to have it say ( Voter's comments )
and move them there after the story posts? It would be less krufty and that way
we could comment on predisscussed "packaging" and leave the actual content
discussions here.

Just a thought, probably already been thunk.

Re: win-win and a suggestion (none / 0) (#20)
by rusty on Tue Jun 06, 2000 at 05:44:20 PM EST

And a win for Free software is a win for everyone. :-) Won't be long before more sites realize that the Free news/discussion systems are usually head and shoulders above the commercial (proprietary) software. Ask the folks over at infoworld's IWE forums what they think of their last commercial BBS system. And while you're at it, ask them why they aren't considering a Free alternative yet.

Your ides about handling moderation comments: I think that actually might be a great way to handle it, with one addition. What if you could mark your comment "writeup-related" or "topic-related", and have topic-related comments go into the normal discussion, and story-related ones go into a special area, like you suggest. I totally agree that several aspects of voting need to be fixed.

Oh yeah, and sorry about the scoop site being down. I'm scrambling for time, but I should be able to fix it tonight.

____
Not the real rusty
[ Parent ]

Re: win-win and a suggestion (none / 0) (#22)
by SgtPepper on Tue Jun 06, 2000 at 05:54:38 PM EST

Your ides about handling moderation comments: I think that actually might be a great way to handle it, with one addition. What if you could mark your comment "writeup-related" or "topic-related", and have topic-related comments go into the normal discussion, and story-related ones go into a special area, like you suggest. I totally agree that several aspects of voting need to be fixed.

If I may be so bold, that just adds more complexity.  It is my opinion
that the comments made in the story moderation queue shouldn't
directly relate to the story, but with the system the way it is
you don't have much choice.  I mean it almost makes it too easy.
Now if people knew that any comments they made there directly
related to the editorial value of a submission, then they might
think a bit harder about how it should be, I mean, look at the story
we had earlier about crypto, it went through numerous rewrites 
before being posted, but some people made comments in there that
are now gone.  If the two functions are seperated it makes things
easier on everyone.  

At least that's what I think :)

*goes off to download the CVS of scoop to see what else he can do*


[ Parent ]
Shockwave at eScout (none / 0) (#17)
by KindBud on Tue Jun 06, 2000 at 03:43:48 PM EST

What's the deal with the shockwave fluff at eScout? No shockwave for Solaris...

Doesn't seem to be necessary to navigate the site, but doesn't make a great first impression to have to cancel Netscape's search for the non-existent plugin.

--
just roll a fatty

anyone else can't get to scoop? (none / 0) (#25)
by abe1x on Tue Jun 06, 2000 at 10:26:29 PM EST

Am I I the only one who can't get to - http://scoop.kuro5hin.org/ - I've been trying for several days and keep getting "Internal Server Error"

Re: anyone else can't get to scoop? (none / 0) (#26)
by rusty on Wed Jun 07, 2000 at 02:07:26 AM EST

Um, no, you're not the only one. It's broken for everyone. But you know that when you go there, it's broken special for you. :-)

Seriously, I just Have Had No Time (#273 in the Eternal Litany of Lame Excuses, for those keeping track) to fix it. It will be fixed soon.

____
Not the real rusty
[ Parent ]

Re: anyone else can't get to scoop? (none / 0) (#27)
by Inoshiro on Wed Jun 07, 2000 at 02:10:18 AM EST

Let's just say it was a case of massive hardware failure.

In this case, the power went out, and some data became really out of sync.. Brent, the guy who kindly hosts it for us, is working hard to get the site back up. You might also note that the site he runs, geeky.org, is also down :-(



--
[ イノシロ ]
[ Parent ]
Scoop In a Production Environment | 30 comments (30 topical, 0 editorial, 0 hidden)
Display: Sort:

kuro5hin.org

[XML]
All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!