sorry, forgot to put my email: thegoodlife@comcast.net where do you put the robots.txt file how do you code it?

Or set a robot trap - http://www.braemoor.co.uk/software/robottrapphp.shtml

Also using 213.139.94.68

I own a search engine script and it was searching and crawling search results from my site for a couple of unknown mp3 websites.

This bot does not follow robots.txt. Here are their full IP ranges:  38.99.13.112 - 38.99.13.127 64.1.215.160 - 64.1.215.167 208.36.144.0 - 208.36.144.127

this bot is NOT obeying my robots.txt!!!  "User-agent: BecomeBot Disallow: /"  this doesn't have any affect....=(

not sure where you get the 'coming soon' from...the page is clearly labelled as a service affiliated with the Univ of Illinois...

220.181.19.176 - - [17/Jul/2007:09:56:32 +0000] "GET /robots.txt HTTP/1.1" 200 117 "-" "Sogou web spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 220.181.19.176 - - [17/Jul/2007:10:40:45 ...

http://www.omni-explorer.com/   Its user agent string: OmniExplorer_Bot/6.70 +http://www.omni-explorer.com) WorldIndexer.  Visited from IP 70.87.196.242  Made too many requests to my ...

This is an Asian search engine: http://www.baidu.com/  IP used: 122.152.128.48 User Agent String: Baiduspider+(+http://www.baidu.com/search/spider_jp.html)

IEAutoDiscovery seems to be a prefetcher. It is very aggressive and makes a few requests per second. Does anybody have more details about this bot?

Visvo claims that VisBot is a Nutch based bot that crawls websites "in preparation for a new type of search software". The full user agent string for this bot is "VisBot/2.0+(Visvo.com+Crawler;+http://www.visvo.com/bot.html;+bot@visvo.com)". ...

Seems to be another spider for sogou.com. Other two are:  - Sogou Spider http://adminter.net/User-Agent.aspx/sogou%20spider - Sogou Web Spider http://adminter.net/User-Agent.aspx/Sogou%20Web%20Spider ...

I've never seen "Qihoo DSP-1.2" bot before. Today it came to one of my sites and made a few requests from the 211.100.25.70 and 211.100.25.76 IP addresses without checking robots.txt. ...

Mediapartners-Google is the Google AdSense bot. You should see requests from this crawler only if you have AdSense on your web site. All the requests must come from Google IP addresses.

I've seen "Solatis Internet Spider" on a couple of my sites. It comes from 82.75.133.7 and crawls very fast. It send just one HTTP header "User-Agent: Solatis Internet Spider/0.1".

"Sogou Web Spider" a crawler for sogou.com. It comes from the same IP addresses as "Sogou Spider" and probably is the same bot that uses different user agent strings. The full user ...

ConveraCrawler is a bot developed by convera.com. It seems they are selling gathered information to third parties. The full user agent string for this bot is "ConveraCrawler/0.9e (+http://www.authoritativeweb.com/crawl)". ...

NutchCVS is a web crawler for the Nutch open source search engine. For more details about this bot see http://adminter.net/User-Agent.aspx/Nutch

Lately I am seeing Nutch requests from two IP addresses 75.126.142.100 and 72.232.228.58. Both of them uses same user agent string "Nutch Test/Something strange (Nutch Test; http://www.mistral.com)". ...

I've seen LarbinWebCrawler coming from two IP addresses 213.114.34.219 and 213.114.34.232. It uses either "LarbinWebCrawler spider@download11.com" or "LarbinWebCrawler internet@bredband.net" ...

Googlebot is a web crawler used by the Google search engine. It used to have "Googlebot/2.1 (+http://www.google.com/bot.html)" user agent, but lately it switched to "Mozilla/5.0 (compatible; ...

Wget is a content retrieval application and is part of the GNU project. It supports HTTP, HTTPS, and FTP protocols, however it does not seem to support HTTP compression. Also Wget supports ...

I could not find who owns this bot. nicebot does not send any contact info in the HTTP headers. It comes from the following IP addresses:  serverpronto.com addresses: 64.251.30.20 64.251.30.21 ...

Here are some of IP addresses that recently made requests to pages on my sites using larbin:  User-Agent:   larbin_2.6.3 larbin2.6.3@unspecified.mail 60.28.2.109 64.62.187.6 66.80.248.144 ...

I've seen QihooBot requests from all the IP addresses in the 220.181.34.161-220.181.34.190 range. It is a crawler for a Chinese search engine http://www.qihoo.com. The full user agent ...

voyager/1.0 seems to be just another name for cfetch/1.0 (http://adminter.net/User-Agent.aspx/cfetch). Both of them come from 38.113.234.180 and 38.113.234.181 IP addresses. I am not ...

I've seen cfetch/1.0 on all my web sites. This bot comes from 38.113.234.180 and 38.113.234.181 IP addresses. It seems to be a crawler for kosmix.com (they send "From: cosmix@cosmixcorp.com" ...