Google has been undergoing a lot of big changes during the last couple months, though, and things have been quite odd to say the least. A lot of people seem to think Google is moving toward a rolling update instead of a re-ranking of the entire web once a month.
The weirdness started in May. The monthly update (Dominic) was late - very late. The Deepcrawler, which spidered sites deeply once a month for the big update, hadn't been seen in the logs of anyone who watched for it. Very strange. Forums across the 'net experienced a flurry of activity as people tried to figure out what was going on at the Googleplex.
To add to the mysteriousness, Freshbot began to act a little more like the Deepcrawler. Traditionally, Freshbot came from a different IP range and added pages to the index immediately as opposed to once a month like Deepbot did. Freshbot's purpose was to add 'freshness' to the search results in between monthly updates.
Did the late monthly update and disappearance of Deepbot signal the birth of a new bot? Names like deepfreshBot and FredBot were posted around the 'net as people pondered the new activity in their logs.
GoogleGuy (a Google employee who posts anonymously at WebmasterWorld - yes, it's been verified) confirmed that a big change was in the works and that people should be patient as Google evolved. He said the process would take "more than weeks, less than months."
Cryptic, but to be honest, I'm happy to get any first-hand, 'unofficial' info from them. During Dominic, SERPs (search engine result pages) were all over the place. Some pointed back to the now infamous September '02 update that was talked about in Wired and on that other site.
The doomsayers always appear during an update, though, predicting the downfall of the 'obviously-evil-because-they-don't-list-my-site' Google. The conspiracy theorists come out in droves as well.
One theory that was offered to the world was that Google had run out of unique id's because they used four bytes instead of five or more when designing their database infrastructure. Googleguy mentioned that a Google employee fell out of his chair laughing when he heard that.
Beyond the anger and conspiracies, though, some took GoogleGuy's messages and tried to make sense of what was going on. (This whole marketing thing Google has going on with GoogleGuy may be a story in itself.)
Anyway, although no one from Google has officially confirmed it, they're apparently moving away from a once-a-month crawl/update cycle to a more continuous updating process. As the web grows, this is becoming a necessity to have the freshest results possible. They're also perfecting (as much as they can) automated spam detectors to help clean up the SERPs.
For Google to stay on top (as they edge closer to an eventual IPO), these are worthy tasks. They need to innovate. They need to stay two steps ahead of their closest competitors, even if it means a little collateral damage in the SERPs for a little while. At this time, in my humble opinion, the only SE that comes close to rivaling Google is FAST's AllTheWeb.
Yahoo is most likely going to dump Google as a provider of their search engine results at some point this year, though. And Microsoft is also showing an interest in the search engine market. With two giants like that (not to mention FAST becoming a giant in its own right), it's becoming more and more important for Google to maintain their lead.
As I said in the intro, the Esmeralda update appears to be settling down somewhat and should stabilize sometime in the next few days. From there, SEOs (search engine optimizers) will again look at how Google ranks the web and how best to optimize for it.
The bigger question, though, is what it means for the Internet at large? Personally, I think it's an effort by Google to maintain their dominance in the search engine game. And Google has always been a good neighbor in cyberspace so this is not necessarily a bad thing.
Sure, some may snort and chuckle when Microsoft and search engines are mentioned in the same sentence, but they have a *lot* of money. In our society this means they don't necessarily need the best product to have the most market share. Just take a look at Internet Explorer and its dominance over Netscape and other browsers.
The thing is, if Google can remember what they were like when they were still the little guys, they have a fighting chance at fending off Microsoft and the others. This latest major change to their algorithm and crawling methods is, I think, a move by Google to sprint ahead of the competition before it's too late.
Go, Google, go.