Fergie's Tech Blog

Be The Bot: Spoofing Googlebot

Sean Michael Kerner writes on internetnews.com:

It crawls the Web without malice seeking out every possible bit of content. It's name is Googlebot, and sometimes it gets to see things on the Web that the rest of don't.

Unless of course you pretend to be Googlebot.

Superficially spoofing Googlebot, Google's Web crawler, is not a difficult thing to do and was recently the subject of a very popular post on the Digg site. Since at least September of 2006 however, Google has made efforts to help webmasters protect themselves against spoofed Googlebots. That doesn't mean people still aren't trying to be Googlebot, if the popularity of the Digg post is any indication.

posted by Fergie @ 7/17/2007 11:50:00 AM