User Agents > googlebot
googlebot
Rating: 0
Mike - 3 Apr 2007 5:09 AM
Mike - 3 Apr 2007 5:09 AMGooglebot is a web crawler used by the Google search engine. It used to have "Googlebot/2.1 (+http://www.google.com/bot.html)" user agent, but lately it switched to "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)". Googlebot supports robots.txt exclusion standard and it also can be blocked using robots meta in the HTML header.
Sometimes content scrapers also use the Googlebot user agent string. Here are a few non-Google IP addresses that crawled some of my websites using "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)".
72.36.153.26
72.232.225.218
87.118.112.15
Sometimes content scrapers also use the Googlebot user agent string. Here are a few non-Google IP addresses that crawled some of my websites using "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)".
72.36.153.26
72.232.225.218
87.118.112.15
Reply