Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

Biological Webcrawlers

By ikarus in Internet
Mon May 21, 2001 at 09:46:00 AM EST
Tags: Internet (all tags)

Possibly the most powerful distributed search engine on the internet is the network of human beings that make it up. Sure it may be the slowest search around, but it yields surprisingly relevant results. It's no good if you're looking for information on dancing lemurs, but it can't be beat if your looking for places to hang out and spend your time. How does this wonder-indexer work? I don't know, and I was hoping we could figure that out.

I've been thinking a lot lately about how exactly the internet functions from a human standpoint. There are a number of ways for raw information to be transferred, but the most important, and valuable, information seems to travel between people. This information usually seems to be of the recommendation type, meaning it's experience related information. In a way it's not unlike the passing of generational knowledge from one group to the next.

What exactly am I talking about? Well, I'll go out on a limb and use myself as an example. When I first went online I probably knew about less than a dozen sites on the internet. I knew that if I wanted to find stuff, I could probably go to Yahoo! or WebCrawler, and I could find my way to a few corporate sites for information about things I was interested in. Aside from that I didn't know much. However, over time, I came to learn of AltaVista (when it was brand new), and eventually Google (currently all the rage). I don't really know how (under what influence) I made those transitions, but I did. In some manner, some one passed along knowledge about where I could go to find what I was looking for.

The same is true for many of the sites I visit and hang out at today. I don't know how, why, or under what circumstance, but I found my way to Slashdot, to Kuro5hin, and many other sites that I enjoy and frequent. One thing that I can say for sure though, is that I have never in my life seen an ad for Google, Slashdot, freshmeat or Kuro5hin. It seems as though the recommendation, the knowledge, experience, or whatever you choose to call it came to me via another person, and that it also came to them via another person.

It seems as though the best form of advertising and promotion is none at all. The key to success, seems to lie in initially connecting with a solid and vigilant user base that spreads the word. I can't say I've ever seen an ad for Napster. Many of today's popular sites, seem to have grown mainly, even exclusively, through word of mouth.

The particularly interesting thing to me has been the introduction of information dissemination points. Take Slashdot for example. They have essentially gained the power to influence a large community by choosing to plug or link other sites. I think this is evidenced by the relative success of sites that have been mentioned there, particularly their own (newsforge, sourceforge). The "slashdot effect" has a fear/crave association to it. Is this power or ability good or bad?

So really, I would like to hear what all of you have to say on the subject. How do you hear about sites? How did you come to k5? If you have ever created a site, how did you get users? Was it slow and steady, was their a bolt of registrations? Do you even agree with some of what I have theorized above? And, what is your take on how information travels on the internet?

(Oh, and it you have any good sites on dancing lemurs, I'd love to know about them.)


Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure


Related Links
o Slashdot
o Kuro5hin
o Google
o Freshmeat
o Also by ikarus

Display: Sort:
Biological Webcrawlers | 15 comments (15 topical, editorial, 0 hidden)
Info flow (3.00 / 1) (#1)
by dzimmerm on Fri May 18, 2001 at 05:18:04 PM EST

I heard about K5 on slashdot.

I learned about slashdot from some friends who were more into weblogs than I was at the time.

Before the web became popular I helped to run a BBS system that had quite a few users. Once the WWW started going strong I took the BBS down as I felt it was redundent.

I would agree that word of mouth is a starting point. Once you get a few search engines under your belt and a few weblogs as part of your dailey routine you get more info than you can manage.

+1 to section


eh (4.00 / 1) (#2)
by delmoi on Fri May 18, 2001 at 06:38:25 PM EST

I'm surprised you don't know why you come to these sites, for it's easy to remember what started. And since the other poster talked about how he got to k5, I figure I might as well to... I was here when kuro5hin was 'born' in the threads of slashdot, when people really wanted to be able to see the queue themselves. But, when I first started checking out the site most of the comments were only one or two lines long(!). Since there wasn't that much here, I left. And eventually I came back later, and was surprised to see a flourishing site.

I got to slashdot, actually, by way of Jon Katz's articles for Wired online. At the time he seemed like an interesting read...

Wired online, of course, came from reading wired. I really don't know why I got into that magazine, probably an issue had a cool cover or something... but I knew it was about 'internet culture' or something like that, and at the time that really fascinated me (not having a connection of my own, I was like 15 or so).
"'argumentation' is not a word, idiot." -- thelizman
How I came to kuro5hin (4.33 / 3) (#3)
by John Milton on Fri May 18, 2001 at 08:05:17 PM EST

I got interested in linux, and all of my websearches for information kept leading back to slashdot. I read that for a little while, because I was kinda nervous about setting up an account and talking to people I'd never met. Obviously, I'm over that now. :)

I'd heard about kuro5hin on slashdot. I was never too interested until I figured out that you could submit your own stories and vote on others. Also, I nearly cracked up when I saw the bbspot's parody. What really made me leave slashdot was the Michael. It's pretty obvious that he uses his infinte mod points to censor anything that puts him in a bad light.

You guys are more egalitarian, and that's what I'm looking for. You make me feel warm and fuzzy inside. *sniff sniff* Can I have a tissue?

"When we consider that woman are treated as property, it is degrading to women that we should Treat our children as property to be disposed of as we see fit." -Elizabeth Cady Stanton

blah (3.00 / 1) (#4)
by regeya on Fri May 18, 2001 at 10:13:39 PM EST

found out about k5 from /.

fount out about /. not long after it started because of AfterStep and Enlightenment. Rob Malda wrote a neat little thing called ePlus.

found out about Enlightenment and AfterStep through this site a few years back, looking for a suitable windowmanager, and because of my interest in Linux.

Got involved with using Linux because of a systems programming course in college.

So, you can blame college for my worthless blog addiction.

[ yokelpunk | kuro5hin diary ]

Little question? (3.00 / 1) (#5)
by John Milton on Fri May 18, 2001 at 10:43:22 PM EST

I hate to reveal ignorance, but why are they called blogs? What does the b stand for?

"When we consider that woman are treated as property, it is degrading to women that we should Treat our children as property to be disposed of as we see fit." -Elizabeth Cady Stanton

[ Parent ]
Blogs (3.00 / 1) (#6)
by Wolfbaine on Fri May 18, 2001 at 11:08:21 PM EST

Blogs are shorthand for weB log.

[ Parent ]
Picking out strands of the web (3.00 / 1) (#7)
by Kellnerin on Fri May 18, 2001 at 11:12:26 PM EST

This is not a "How I Came to K5" comment; I'm actually not sure why I started frequenting the site, only that I keep coming back, which must be a good thing.

What I do know is it probably did come up in conversation at one point, probably with a friend of mine: he reads /. and The Register, I read k5 and Salon. We each know what the other is interested in, and trade links at the end of the day. Word of mouth, or some kind of referral, is powerful. Arts & Letters Daily bills itself as exactly the kind of thing you're talking about: "A search engine that shows you things you weren't looking for." It's all Link Propagation, though usually not so Mindless, and more culture than technology. It's not as up-to-the-minute as Slashdot, doesn't have the volume of new postings of most places, and I doubt there's an "ALDaily" effect, but it will point you to some strange, and often interesting places ...

--Stop it, evil hand, stop it!--

Media Virus (4.50 / 2) (#8)
by Duke Machesne on Fri May 18, 2001 at 11:21:39 PM EST

For some coherent thoughts on the viral spread of ideas and (in the case of the web) directions, I highly recommend Douglas Rushkoff's almost ridiculously optimistic book Media Virus! and his more sober, less technocentric follow-up Coercion. Check out his disinfo dossier.

arts schoolsweight loss

I suck (4.25 / 4) (#9)
by anonymous cowerd on Sat May 19, 2001 at 06:41:27 PM EST

...If you have ever created a site, how did you get users?

If you search in Yahoo! or Google for "Schopenhauer" you will find my transcriptions of English translations of some of his essays. I am in the middle of slowly grinding out a transcription of the (now public-domain) Haldane/Kemp translation of The World as Will and Idea, a huge piece of work (I am attempting to provide translations for the zillion or so Greek, Latin, French, etc. quotes in there). I must get a good hundred page-hits a year off that material. Just today I got a nice email from a guy in Brazil who downloaded my stuff. Made my day!

...How did you come to k5?

Well, when the Columbine fun twins went on their world-famous rampage, I was looking in the sidebar to the Yahoo! news article on it. Among the innumerable various blanket indictments of Quake players, non-Christians, people who prefer to dress in black, users of the Internet, KMFDM fans, teenagers in general, prettty much every cultural minority that any stuffy punditorial jackass felt any urge to denigrate, there stood out an unusually - uniquely! - sympathetic column by one Jon Katz. Almost as though the trenchcoat two were, like, human, bent sure, yet there but for the grace of God go we, rather than being opaque, alien villains akin to Fredric Brown's clicking clawed roller-ball.

Clicky clicky, next thing you knew I had stumbled upon the mighty and glorious slashdot, which was even better than Usenet in that, while I publicly emit blather that everyone disregards, it allows me to use bold and italics. Wow, to my surprise and delight, the Internet has other uses besides driver downloads and scanned photos of undressed women. slashdot leads to K5, K5 leads weissnichtwo, guess I'll find out.

See, having finally given up in my old age on friendship or even respect, I now have no fear of your inevitable and doubtlessly richly-deserved disdain! Jon Katz brought me here. Hey, aside from its overly-corny title (something about "Hellmouth," if that word had a prior usage I missed it), I thought that article of his was pretty decent. Sue me.

Yours WDK - WKiernan@concentric.net

stint grits
darts file
gratis ways to fit tins
dapper angle
ill apple
-Barbara Baracks

I'm different (4.00 / 1) (#10)
by Dacta on Sun May 20, 2001 at 01:53:06 AM EST

I heard about Slashdot from.. umm.. I don't know actually. It was a long time ago.

I came to first came to kuro5hin after reading some discussion on the Slash-code mailing list about how long it was taking to get a new release of Slash from the slashdot people. Rusty was annoyed, and told everyone he was writting something called Scoop, which was essentially a better version of Slash. It sounded interesting, so I had a look at Kuro5hin. K5 was still based on Slash then and was pretty boring (there were abtou 50 users). Then one day there was some story I wanted to comment on, and I didn't want to be an Anonymous Hero (this was back when they were still around), so I signed up. That was in March (?) 2000, I think.

not an old timer... (4.00 / 1) (#11)
by univgeek on Sun May 20, 2001 at 07:59:13 PM EST

I dont seem to be an old-timer, unlike those who have already posted comments.

I think word of mouth seems to be one of the best drivers of traffic. I had heard of Google from a friend of mine and I soon had quite a few people hooked on it. I'm still doing that with /. and k5 :-).

Also I think newspapers would play a big role for those who are just getting on the net. Most papers seem to have columns on how to use the net 'usefully'. I have gone to quite a large number of web-sites which were talked about in the papers.

And yeah I seem to be frequenting K5 quite a bit now.
Arguing with an Electrical Engineer is liking wrestling with a pig in mud, after a while you realise the pig is enjoying it!

My journey... (4.00 / 1) (#12)
by Office Girl the Magnificent on Mon May 21, 2001 at 09:33:30 AM EST

I came to K5 independant of /. I never read /. I mean, I'd seen it, read an article or two at my sweetheart's behest, but I just couldn't get into it. Too much crap to wade through.

Then, above mentioned sweetheart (PedroPicasso) suggested I look at this K5 thing. "It's like /.," he said, "but with culture." So I came here, created an account, and I've been hooked. What I love is that this site acknowledges that there is more than one kind of geek.

Given my experience, I'd say word of mouth is extremely important -- whether that word is through people you know personally or people you know via the web. This can even include weblinks from a personal page -- a webmaster's reccomendation is important to me, especially if I like the reccomending site.

And if you like this post, be sure to click on my .sig for more! :)

"If you stay, Infinite might try to kill you. If you leave, the FBI definitely will. And if you keep yelling, I might do it myself."

Me too (5.00 / 1) (#13)
by Tatarigami on Mon May 21, 2001 at 07:42:54 PM EST

How do you hear about sites?

I keep an eye on a few specialised portals:


Both of them allow me to laugh at the chronically stupid. I'm not a nice person. Other than that, I bug my friends for links. God knows where they find some of them, but they make a good quality filter.

How did you come to k5?

Found a link to it in a sig on a /. post. Haven't been back there since...

If you have ever created a site, how did you get users?

Had the users before the site. I had my audience in mind before I started writing HTML. One section was for my workmates, who provided the content as well as the audience, the other was for members of a club I joined.

Was it slow and steady, was their a bolt of registrations?

Slow and steady, based on the feedback I've received. I don't have a counter or bother with site statistics, so I'm not sure I'd notice a bolt.

Do you even agree with some of what I have theorized above?

Yup. Search engines are good for immediate results, but they're not smart enough to judge the difference between genuine (entertaining!) content, and the millions of indistinguishable sites out there. That takes human judgement, and even bad human judgement is an improvement over what a search engine can do.

And, what is your take on how information travels on the internet?

By boat.


Link topology of social networks (none / 0) (#14)
by KWillets on Fri Jun 01, 2001 at 08:28:41 PM EST

There have been a number of studies of the link topology of the web, and how it reflects the structure of social networks in general.

The web contains billions of pages, yet an individual page will have an average of only ten links or so. Researchers quickly found that these links are highly non-random. Web pages often fall into cliques of mutually-referring "communities", where a small number of pages link to each other more often than to other pages. This structure reflects the fact that webpage authors frequently copy links and sets of links from other pages they know, and pages on a common topic often link to each other.

This type of network is called a social network because it mimics the friend-to-friend connections of human society. People often find friends through other friends, and end up in groups where everybody knows each other. Even things like citations in scientific research papers tend to follow the same pattern. (In fact, Google's pagerank algorithm was based on citation analysis studies from 30-40 years ago.)

There are some other studies about the strength of web pages' connections to the web as a whole. Most newer pages start out with few people linking to them, and only eventually become "strongly connected", where they are only a few hops away from many, many other pages.

Good point (none / 0) (#15)
by vla1den on Mon Jun 11, 2001 at 12:15:51 AM EST

Search for information is probably one most important thing people doing online. And it seems that most effective ways to find quality resources are: "word of mouth", "link in a sig on a /. post", "though the recommendation, the knowledge, experience, or whatever you choose to call it came to me via another person, and that it also came to them via another person".
So what about ads, search engines, or directories? It seems that the trust (and quality) of the reference is proportional to how much opinion of "another person" is represented in it. That is why ads are not trustworthy at all, search engines are neutral, and directories are somehow good (but not as representative as search engines). BTW, I believe that Google is so good because it actually automatically uses human (webmaster) opinion on how good some particular page is. I wonder why nobody else uses this idea. I wrote some system that actually doing exactly this - ranking and categorizes pages by visitors input. And I believe that ultimately the approach of collective knowledge formalization & structuring is something that can be extremely important for the future of Internet.

...what the next best thing is going to be?
Biological Webcrawlers | 15 comments (15 topical, 0 editorial, 0 hidden)
Display: Sort:


All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!