A6\-Indexer
ADmantX
API[\+\s]scraper
AddThis
Alexandria(\s|\+)prototype(\s|\+)project
AllenTrack
Arachmo
ArchiveTeam
BDFetch
BUbiNG
Betsie
BingPreview
Blackboard[\+\s]Safeassign
Brutus\/AET
CakePHP
China\sLocal\sBrowse\s2\.6
Code\sSample\sWeb\sClient
ColdFusion
ContentSmartz
CoverScout
DSurf
DTS Agent
DataCha0s\/2\.0
Demo\sBot
DeuSu\/
Dispatch\/\d
Docoloc
Download\+Master
EBSCO\sEJS\sContent\sServer
ELinks\/
EThOS\+\(British\+Library\)
EasyBib[\+\s]AutoCite[\+\s]
EmailSiphon
EmailWolf
Embedly
FDM(\s|\+)1
FDM(\s|\+)\d
FeedFetcher
Fetch(\s|\+)API(\s|\+)Request
Fulltext
Funnelback
G-i-g-a-b-o-t
GLMSLinkAnalysis
Genieo
GetRight
Goldfire(\s|\+)Server
Googlebot
Grammarly
HTTPFetcher
HTTrack
HttpComponents\/1.1
Indy Library
LOCKSS
LWP\:\:Simple
LinkLint-checkonly
LinkTiger
LongURL.API
MSNBot
MarcEdit.5.2.Web.Client
MetaURI[\+\s]API\/\d\.\d
Microsoft Office Existence Discovery
Microsoft Office Protocol Discovery
Microsoft(\s|\+)URL(\s|\+)Control
Microsoft-WebDAV-MiniRedir
Milbot
MuscatFerre
NABOT
NaverBot
Ning
Offline(\s|\+)Navigator
OurBrowser
PHP\/
Pcore\-HTTP
PycURL
Python\-urllib
Qwantify
Readpaper
Riddler
Scrapy\/\d
SearchBloxIntra
SkypeUriPreview
Sogou\sweb\sspider/4.0
Strider
Sysomos
T\-H\-U\-N\-D\-E\-R\-S\-T\-O\-N\-E
Teleport(\s|\+)Pro
Teoma
The\+Knowledge\+AI
Trove
URL2File
WWW\-Mechanize
Wanadoo
Web(\s|\+)Downloader
WebCloner
WebCopier
WebReaper
WebStripper
WebZIP
Webinator
Webmetrics
Wget
Xenu(\s|\+)Link(\s|\+)Sleuth
[+:,\.\;\/\\-]bot
[^a]fish
^.?$
^@ozilla\/\d
^Array$
^FOCA
^FileDown$
^Filter$
^IDA$
^LinkParser\/
^LinkSaver\/
^MSIE
^Mozilla$
^Mozilla.4\.0$
^Mozilla.5\.0$
^Mozilla\/4\.0\+\(compatible;\)$
^Mozilla\/4\.0\+\(compatible;\+ICS\)$
^Mozilla\/4\.5\+\[en]\+\(Win98;\+I\)$
^Mozilla\/5.0(\s|\+)Gecko\/20100115(\s|\+)Firefox\/3.6$
^Mozilla\/5.0\+\(compatible;\+MSIE\+6\.0;\+Windows\+NT\+5\.0\)$
^Mozilla\/5\.0\+like\+Gecko$
^NetAnts\/\d
^Opera\/4$
^Postgenomic(\s|\+)v2
^Traackr\.com$
^\%?default\%?$
^firefox$
^integrity\/\d
^java\/\d{1,2}.\d
^oaDOI$
^okhttp$
^ruby$
^scrutiny\/\d
^undefined$
^unknown$
^user.?agent$
^voltron$
^voyager\/
^破解后的$
^脝脝陆芒潞贸碌脛$
acme\.spider
alexa
almaden
appie
architext
archive\.org_bot
aria2\/\d
arks
asterias
atomz
autoemailspider
awbot
baidu
baiduspider
bbot
biadu
biglotron
binlar
bjaaland
blaiz\-bee
bloglines
blogpulse
boitho\.com\-dc
bookmark\-manager
bot
bot[+:,\.\;\/\\-]
bspider
bwh3_user_agent
celestial
cfnetwork
cfnetwork|checkbot
checklink
checkprivacy
cloakDetect
coccoc\/1\.0
collection@infegy.com
com\.plumanalytics
combine
commons\-httpclient
contentmatch
convera
core
crawl
crawler
curl\/
cursor
custo
daumoa
docomo
dtSearchSpider
dumbot
easydl
exabot
facebookexternalhit\/
fast-webcrawler
favorg
feedburner
feedfetcher\-google
feedreader
ferret
findlinks
findthatfile
gaisbot
geturl
gigabot
girafabot
gnodspider
google
grub
gulliver
gvfs\/
harvest
heritrix
hl_ftien_spider
holmes
htdig
htmlparser
http.?client
httpget
httpget\-5\.2\.2
httpget\?5\.2\.2
httrack
iSiloX
ia_archiver
ichiro
iktomi
ilse
internetseer
intute
iskanie
java
java\/
jeeves
jobo
kyluka
larbin
libcurl
libhttp
libwww
libwww\-perl
lilina
link.?check
linkbot
linkcheck
linkchecker
linkscan
linkwalker
lipperhey
livejournal\.com
lmspider
ltx71
lwp
lwp\-request
lwp\-tivial
lwp\-trivial
lycos[\_\+]
lycos[_+]
mail.ru
mediapartners\-google
megite
milbot
mimas
mj12bot
mnogosearch
moget
mojeekbot
momspider
motor
msiecrawler
msnbot
myweb
nagios
netcraft
netluchs
ng\/2\.
no_user_agent
nomad
nutch
ocelli
onetszukaj
panscient
parsijoo
pear.php.net
perman
pioneer
playmusic\.com
playstarmusic\.com
powermarks
proximic
psbot
python
qihoobot
rambler
redalert
redalert|robozilla
robot
robots
robozilla
rss
scan4mail
scientificcommons
scirus
scooter
seekbot
seznambot
shoutcast
slurp
sogou
speedy
spider
spiderman
spiderview
summify
sunrise
superbot
surveybot
tailrank
technoratibot
titan
turnitinbot
twiceler
ucsd
ultraseek
urlaliasbuilder
urllib
validator
virus.detector
virus[_+]detector
voila
voyager\/
w3af.org
w3c\-checklink
webcollage
weblayers
webmirror
webmon
webreaper
wordpress
worm
www\.gnip\.com
xenu
y!j
yacy
yahoo
yahoo\-mmcrawler
yahoofeedseeker
yahooseeker
yandex
yodaobot
zealbot
zeus
zyborg