Namely, that it was an order of magnitude more likely that the problem was on your side rather than Google's;
that the Russian Mafia were almost certainly not involved;
that the use of Webmaster Tools would have quickly shown what Googlebot was trying to fetch (and why);
that Googlebot was still fetching deadlinks (the parking pages), despite your insistence otherwise ("It's not just following bad links. If that were the case, it would have given up by now");
that Googlebot does not always identify itself with a useragent string, since that string is widely abused by spammers.
Things I got dead wrong:
Assuming the world's greatest debugger wouldn't fuck up an Apache config, or would at least consider the possibility before fingering a cross-border underworld plot;
not having the necessary psychic skills to find out that you hadn't signed up to Webmaster Tools, (the Googlebot-debugging equivalent of gdb, and something which you'd expect someone priding themself on their SEO and webmaster skills to know about), information I should have picked on telepathically since you never once pointed it out, despite my repeating the suggestion to use WMT ad nauseum;
assuming that the problem was with softwareproblem.net, where you said it was, and not with some other domain.
If I was in your position, with a fucked up Apache config and wrongly thinking the problem was with softwareproblem.net, I would consider the most likely options first: perhaps the problem is on my side. Perhaps I should try this tool that shows my what the Googlebot is seeing. Perhaps I should do a few searches to find out why the world leaders in webcrawling might not be using a useragent string, even though I have this belief that they always should. Perhaps I should consider that I'm hosting a few domains on the one VPS, and think about problems I might have introduced there.
But no, that would take a little humility, and recognition of your own fallibility.
So instead, you act like a fucking idiot, and decide that Google has been infiltrated by Russian/Chinese/Eastern European mobsters.
[ Parent ]