世界著名搜索公司的爬虫清单

SightQuestBot/=sightquest.com
Spider_Monkey/=spidermonkey.ca
Surfnomore Spider v1.1=surfnomore.com
Robot@SuperSnooper.Com=supersnooper.com
teoma_agent1=teoma.com
teoma_admin@hawkholdings.com=teoma.com
Teradex_Mapper=mapper.teradex.com
mapper@teradex.com=mapper.teradex.com
ESISmartSpider=travel-finder.com
Spider TraficDublu=traficdublu.ro
Tutorial Crawler=tutorgig.com
UK Searcher Spider=uksearcher.co.uk
Vivante Link Checker=vivante.com
appie=walhello.com
Nazilla=websmostlinked.com
www.WebWombat.com.au=webwombat.com.au
marvin/infoseek=webseek.de
marvin-team@webseek.de=webseek.de
MuscatFerret=webtop.com
WhizBang! Lab=whizbanglabs.com
ZyBorg=wisenut.com
WIRE WebRefiner=wire.co.uk
WSCbot=worldsearchcenter.com
Yandex=yandex.com
Yellopet-Spider=yellowpet.com
Iron33=verno.ueda.info.waseda.ac.jp/
ALink=Link Checkers
AMeta=Link Checker
ASPSearch URL Checker=Link Checker
BlogBot=Link Checker
BMChecker=Link Checker
Bookmark Buddy=Link Checker
Check&Get=Link Checker
CheckWeb=Link Checker
CNET_Snoop=Link Checker
CSE HTML Validator=Link Checker
DRKSpider=Link Checker
DISCo Watchman=Link Checker
DoctorHTML=Link Checker
Email Extractor=Email Extractor
EmailSiphon=Email Extractor
EmailWolf=Email Extractor
FavOrg=Link Checker
Favorites Sweeper=Link Checker
FreshLinks.exe=Link Checker
Funnel Web Profiler=Link Checker
Html Link Validator=Link Checker
The Informant=Link Checker
The Intraformant=Link Checker
InternetLinkAgent=Link Checker
InternetPeriscope=Link Checker
javElink=Link Checker
jdwhatsnew.cgi=Link Checker
JRTS Check Favorites Utility=Link Checker
Lambda LinkCheck=Link Checker
LinkLint-checkonly=Link Checker
LinkAlarm=Link Checker
Linkbot=Link Checker
Linkman=Link Checker
LinkProver=Link Checker
Links=Link Checker
LinkScan Server=Link Checker
LinkSweeper=Link Checker
Link Valet Online=Link Checker
LinkVerify Spider=Link Checker
LinkWalker=Link Checker
Morning Paper=Link Checker
MoveAnnouncer=Link Checker
NetLookout=Link Checker
NetMechanic=Link Checker
www.elsop.com=Link Checker
NetMind-Minder=Link Checker
NetMonitor=Link Checker
Netprospector JavaCrawler=Link Checker
online link validator=Link Checker
Rational SiteCheck=Link Checker
Robozilla=Link Checker
RPT-HTTPClient=Link Checker
SurfMaster=Link Checker
SyncIT=Link Checker
Watchfire WebXM=Link Checker
WatzNew Agent=Link Checker
WebSite-Watcher=Link Checker
WebTrends Link Analyzer=Link Checker
Weblink Scanner=Link Checker
Xenu's Link Sleuth=Link Checker
W3C_Validator=Link Validator
WDG_Validator/=Link Validator
Tooter=Link Validator
citenikbot/=citenik.co.uk
CLIPS-index=clips-index.imag.fr/
Computer_and_Automation_Research_Institute_Crawler=Research Bot
cosmos=xyleme.com
robot@xyleme.com=xyleme.com
DiaGem/=DiaGem
Digimarc WebReader=digimarc.com
EchO!/2.0=voila.com
FinaleRobot=expressus.com
robot-master@expressus.com=expressus.com
Ideare - SignSite=ideare.com
GentleSpider=research.att.com
Gulper Web Bot=Gulper Web Bot
larbin=Unknown Spider
sebastien.ailleret@inria.fr=inria.fr
ghi@lcs.mit.edu=Unknown Spider
MultiText=MultiText
NEC Research Agent=NEC Research Agent
OntoSpider=OntoSpider
sherlock_spider=sherlock.com.cn