世界著名搜索公司的爬虫清单

Steeler=Steeler
ru-robot=rutgers.edu
0.1_hseo(at)cs.rutgers.edu=rutgers.edu
WebGather=WebGather
xyro=xyro
xcrawler@inria.fr=Unknown Spider
Zao/0.2=Zao
ADSARobot=ADSARobot
AnswerChase=AnswerChase
ASPSeek=ASPSeek
AVSearch=AVSearch
Checkbot=Checkbot
DaviesBot=DaviesBot
deepweb=deepweb.com
GigaBaz=brainbot.com
GigaBazVStheWeb=brainbot.com
crawler@brainbot.com=brainbot.com
Giskard=oralco.com
InternetSeer=InternetSeer
ipiumBot=ipiumBot
InsumaScout=InsumaScout
Katriona=Katriona
LEIA=LEIA
LexiBot=lexibot.com
metabot=metabot
NetCruiser=NetCruiser
NPBot=nameprotect.com
NetZippy=NetZippy
NZBot=navigationzone.com
Opencola=opencola.com
Oxxbot1=Oxxbot
Pansophica=Pansophica
Phoaks=Phoaks
PICgrabber=PICgrabber
PictureOfInternet=PictureOfInternet
erik@malfunction.org=Unknown Spider
PintaSpider=PintaSpider
PolyBot=PolyBot
Squid=Squid
Sqworm=Sqworm
TaWWWantula=TaWWWantula
TeraCrawl=TeraCrawl
TurnitinBot=turnitin.com
UCmore=ucmore.com
UdmSearch=mnoGoSearch
unlostBot=unlost.com
URLBlaze=urlblaze.net
UrlScope=UrlScope
Vagabondo=Vagabondo
vspider=vspider
WAVETools=WAVETools
Webbandit=Webbandit
Webclipping.com=Webclipping.com
webcollage=webcollage
WebCompass=WebCompass
WebGenie=WebGenie
Web Magnet=Unknown Spider
WebMiner=Unknown Spider
Webpush=Unknown Spider
WebSymmetrix=Unknown Spider
webrank=Unknown Spider
webwasher=Unknown Spider
WhosTalking=Unknown Spider
AnzwersCrawl/2.0=Anzwers
fido/1.0 Harvest/1.4.pl2=Planet Search
GAIS Robot/1.0B2=seednet
Googlebot/1.0=Google.com
Gulliver/1.2=Northern Light
Infoseek Sidewinder/0.9=Infoseek
KIT_Fireball/2.0=Fireball
lwp-trivial/1.27=Search 4 Free
Lycos_Spider_(T-Rex)/3.0=Lycos
Scooter/1.0=AltaVista
Scooter/1.0 scooter@pa.dec.com=AltaVista
Scooter/1.1 (custom)=AltaVista
Scooter/2.0 G.R.A.B. X2.0=AltaVista
Scooter/2.0 G.R.A.B. V1.1.0=AltaVista
search.at V1.2=search.at
inktomi=Inktomi Spider
SwissSearch V1.2=SwissSearch
The Informant=The Informant
Ultraseek=Infoseek
WebCrawler/3.0 Robot libwww/5.0a=WebCrawler
WebCrawler-AddURL/2.0=WebCrawler
WiseWire=WiseWire
WiseWire-Alpha-1.0=WiseWire
WiseWire-Alpha-Spider=WiseWire
WiseWire-Alpha12-Spider971219a=WiseWire
WiseWire-Alpha12-Spider(971223a)=WiseWire
WiseWire-HotSpider-1.0=WiseWire
WiseWire-Spider=WiseWire
WiseWire-Spider-1.0=WiseWire
WiseWire-Spider2=WiseWire
WiseWire-Widow-1.0=WiseWire
WiseWire-Widow-1.0r=WiseWire
WiseWire-Widow-1.0-ALPHA12=WiseWire
CherryPickerSE/1.0=Email Extractor
CherryPickerElite/1.0=Email Extractor
Crescent Internet ToolPak HTTP OLE Control v.1.0=Email Extractor
EmailCollector/1.0=Email Extractor
EmailWolf 1.00=Email Extractor
ExtractorPro=Email Extractor
ask jeeves=Ask Jeeves
lycos=Lycos.com
whatuseek=What You Seek
wisenutbot=Looksmart
msnbot=MSN
GigaBlast=Gigablast
Gigabot=Gigablast
archive_org=Archive.org
jeeves=Ask Jeeves
Asterias=Singingfish Spider
Slurp=Inktomi Spider
ZyBorg=LookSmart Bot
baiduspider=Baidu