|
Spider |
Hits |
% of Total |
Visits |
1 |
Bing Spider |
6,521,085 |
23.52% |
11,104 |
2 |
BaiDu Spider |
5,907,785 |
21.31% |
175,074 |
3 |
Yandex Spider |
3,164,910 |
11.42% |
23,186 |
4 |
Yahoo Spider |
1,922,346 |
6.93% |
3,307 |
5 |
Wespe.de Spider |
1,878,179 |
6.78% |
582,263 |
6 |
Google Spider |
1,673,072 |
6.04% |
39,176 |
7 |
Mozilla/5.0+(compatible;+SemrushBot/ 1.2~bl;++http:/ / www.semrush.com/ bot.html) |
1,357,923 |
4.90% |
496 |
8 |
Vegi+bot+(we+follow+your+robots.txt+setti ngs+before+crawling,+you+can+slow+down+th e+bot+by+change+the+Crawl-Delay+parameter +in+the+settings.if+you+have+an+enquiry,+ please+email+to:+abuse-report@terrykylese oagency.com) |
1,265,314 |
4.56% |
61 |
9 |
magpie-crawler/1.1+(U;+Linux+amd64;+en-GB ;++http:/ / www.brandwatch.net) |
914,850 |
3.30% |
920 |
10 |
Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10 _10_1)+AppleWebKit/ 600.2.5+(KHTML,+like+Gecko)+Version/ 8.0.2+Safari/ 600.2.5+(Applebot/ 0.1;++http:/ / www.apple.com/ go/ applebot) |
547,327 |
1.97% |
352 |
11 |
Linguee+Bot+(http://www.linguee.com/ bot;+bot@linguee.com) |
478,253 |
1.73% |
387 |
12 |
Treato-Bot |
414,531 |
1.50% |
732 |
13 |
Mozilla/5.0+(compatible;+MJ12bot/ v1.4.7;+http:/ / mj12bot.com/ ) |
188,276 |
0.68% |
1,374 |
14 |
Mozilla/5.0+(compatible;+DotBot/ 1.1;+http:/ / www.opensiteexplorer.org/ dotbot,+help@moz.com) |
176,291 |
0.64% |
3,053 |
15 |
Mozilla/5.0+(compatible;+BLEXBot/ 1.0;++http:/ / webmeup-crawler.com/ ) |
174,440 |
0.63% |
969 |
16 |
Mozilla/5.0+(compatible;+MegaIndex.ru/ 2.0;++http:/ / megaindex.com/ crawler) |
150,441 |
0.54% |
10 |
17 |
Sogou Spider |
148,680 |
0.54% |
7,312 |
18 |
Google image crawler |
95,566 |
0.34% |
950 |
19 |
Mozilla/5.0+(compatible;+proximic;++http: / / www.proximic.com/ info/ spider.php) |
73,218 |
0.26% |
65,374 |
20 |
Facebook external hit |
71,119 |
0.26% |
50,743 |
21 |
Mozilla/5.0+(Windows;+U;+Windows+NT+6.0;+ en-GB;+rv:1.0;+trendictionbot0.5.0;+trend iction+search;+http:/ / www.trendiction.de/ bot;+please+let+us+know+of+any+problems;+ web+at+trendiction.com)+Gecko/ 20071127+Firefox/ 3.0.0.11 |
65,723 |
0.24% |
24,556 |
22 |
MSN Spider |
59,929 |
0.22% |
40 |
23 |
Mozilla/5.0+(compatible;+Steeler/ 3.5;+http:/ / www.tkl.iis.u-tokyo.ac.jp/ ~crawler/ ) |
44,202 |
0.16% |
999 |
24 |
robots |
38,026 |
0.14% |
5,766 |
25 |
Mozilla/5.0+(compatible;+SeznamBot/ 3.2;++http:/ / napoveda.seznam.cz/ en/ seznambot-intro/ ) |
33,962 |
0.12% |
738 |
26 |
Yandex Images Spider |
30,194 |
0.11% |
1,960 |
27 |
Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10 _10_1)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 40.0.2214.111+Safari/ 537.36+moatbot |
29,579 |
0.11% |
2,164 |
28 |
Mozilla/5.0+(compatible;+Cliqzbot/ 1.0;++http:/ / cliqz.com/ company/ cliqzbot) |
24,456 |
0.09% |
1,291 |
29 |
R6_FeedFetcher_(www.radian6.com/ crawler) |
24,107 |
0.09% |
130 |
30 |
Mozilla/5.0+(compatible;+MJ12bot/ v1.4.8;+http:/ / mj12bot.com/ ) |
21,605 |
0.08% |
946 |
31 |
istellabot/t.1 |
17,374 |
0.06% |
239 |
32 |
Mozilla/5.0+(compatible;+ZumBot/ 1.0;+http:/ / help.zum.com/ inquiry) |
16,533 |
0.06% |
0 |
33 |
CCBot/2.0+(http://commoncrawl.org/ faq/ ) |
15,870 |
0.06% |
164 |
34 |
Mozilla/5.0+(compatible;+GrapeshotCrawler / 2.0;++http:/ / www.grapeshot.co.uk/ crawler.php) |
15,244 |
0.05% |
11,251 |
35 |
Pinterest/0.2+(+http:/ / www.pinterest.com/ bot.html) |
12,852 |
0.05% |
2,260 |
36 |
Mozilla/5.0+(compatible;+XoviBot/ 2.0;++http:/ / www.xovibot.net/ ) |
12,322 |
0.04% |
1,681 |
37 |
MaxPointCrawler/Nutch-1.10+(maxpoint.craw ler+at+maxpointinteractive+dot+com) |
12,121 |
0.04% |
704 |
38 |
Alexa Spider |
12,070 |
0.04% |
451 |
39 |
Mozilla/5.0+(Windows+NT+6.1;+WOW64)+Apple WebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 50.0.2661.102+Safari/ 537.36;+360Spider |
12,060 |
0.04% |
4,327 |
40 |
Mozilla/5.0+(compatible;+SiteExplorer/ 1.1b;++http:/ / siteexplorer.info/ Backlink-Checker-Spider/ ) |
6,749 |
0.02% |
71 |
41 |
Mozilla/5.0+(compatible;+spbot/ 5.0.3;++http:/ / OpenLinkProfiler.org/ bot+) |
6,026 |
0.02% |
16 |
42 |
DigitalPebble+Crawler/ 1.4-SNAPSHOT+(StormCrawler+Benchmark;+htt p:/ / digitalpebble.com/ ;+crawler@digitalpebble.com) |
5,939 |
0.02% |
1 |
43 |
istellabot/t.1.13 |
4,945 |
0.02% |
66 |
44 |
Mozilla/5.0+(compatible;+linkdexbot/ 2.0;++http:/ / www.linkdex.com/ bots/ ) |
4,734 |
0.02% |
2,021 |
45 |
Mozilla/5.0+(compatible;+Linux+x86_64;+Ma il.RU_Bot/ 2.0;++http:/ / go.mail.ru/ help/ robots) |
4,138 |
0.01% |
1,388 |
46 |
Mozilla/5.0+(compatible;+SemrushBot-BA;++ http:/ / www.semrush.com/ bot.html) |
3,997 |
0.01% |
1 |
47 |
GarlikCrawler/1.2+(http:/ / garlik.com/ ,+crawler@garlik.com) |
3,335 |
0.01% |
234 |
48 |
YisouSpider |
3,268 |
0.01% |
1,469 |
49 |
Mozilla/5.0+(Linux;+Android+6.0.1;+Nexus+ 5X+Build/ MMB29P)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 41.0.2272.96+Mobile+Safari/ 537.36+(compatible;+Pinterestbot/ 1.0;++http:/ / www.pinterest.com/ bot.html) |
2,805 |
0.01% |
1,359 |
50 |
TinEye-bot/0.61+(see+http:/ / www.tineye.com/ crawler.html) |
2,371 |
0.01% |
175 |
51 |
DigitalPebble+Crawler/ 1.5-SNAPSHOT+(StormCrawler+Benchmark;+htt p:/ / digitalpebble.com/ ;+crawler@digitalpebble.com) |
2,315 |
0.01% |
4 |
52 |
Clickagy+Intelligence+Bot+v2 |
2,047 |
0.01% |
994 |
53 |
IAS+crawler+(ias_crawler;+http:/ / integralads.com/ site-indexing-policy/ ) |
1,921 |
0.01% |
639 |
54 |
adbeat_bot |
1,733 |
0.01% |
630 |
55 |
Mozilla/5.0+(compatible;+YaK/ 1.0;+http:/ / linkfluence.com/ ;+bot@linkfluence.com) |
1,728 |
0.01% |
77 |
56 |
Googlebot+(gocrawl+v0.4) |
1,614 |
0.01% |
745 |
57 |
PDF+Drive+Crawler+1.05+(+https:/ / www.pdfdrive.com/ +) |
1,593 |
0.01% |
598 |
58 |
Mozilla/5.0+(compatible;+MojeekBot/ 0.6;++https:/ / www.mojeek.com/ bot.html) |
1,483 |
0.01% |
707 |
59 |
Mozilla/5.0+(Windows;+U;+Windows+NT+5.1;+ en-US;+rv:1.8.1.6)+Gecko/ 20070725+Firefox/ 2.0.0.6+-+James+BOT+-+WebCrawler+http:/ / cognitiveseo.com/ bot.html |
1,483 |
0.01% |
594 |
60 |
Twitterbot/1.0 |
1,361 |
0.00% |
610 |
61 |
Mozilla/5.0+(compatible;+Findxbot/ 1.0;++http:/ / www.findxbot.com) |
1,316 |
0.00% |
24 |
62 |
Mozilla/5.0+(compatible;+Yeti/ 1.1;++http:/ / naver.me/ bot) |
1,270 |
0.00% |
1,221 |
63 |
GetIntent+Crawler+(http:/ / getintent.com/ bot.html) |
1,267 |
0.00% |
908 |
64 |
Mozilla/5.0+(compatible;+DuckDuckGo-Favic ons-Bot/ 1.0;++http:/ / duckduckgo.com) |
1,165 |
0.00% |
425 |
65 |
Domain+Re-Animator+Bot+(http:/ / domainreanimator.com)+-+support@domainrea nimator.com |
1,038 |
0.00% |
3 |
66 |
Treato-Bot/1.0+(TREATO+LTD,+www.treato.co m,+mailto:+botadmin@treato.com) |
1,036 |
0.00% |
69 |
67 |
Mozilla/5.0+(compatible;+coccocbot-web/ 1.0;++http:/ / help.coccoc.com/ searchengine) |
1,032 |
0.00% |
531 |
68 |
Mozilla/5.0+(Linux;+Android+6.0;+CUBOT_NO TE_S+Build/ MRA58K)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 58.0.3029.83+Mobile+Safari/ 537.36 |
980 |
0.00% |
28 |
69 |
Mozilla/5.0+(compatible;+coccocbot-image/ 1.0;++http:/ / help.coccoc.com/ searchengine) |
952 |
0.00% |
234 |
70 |
Wotbox/2.01+(+http:// www.wotbox.com/ bot/ ) |
907 |
0.00% |
4 |
71 |
BoardReader/1.0+(http:/ / spider.boardreader.com) |
840 |
0.00% |
615 |
72 |
RSSingBot+(http://www.rssing.com) |
831 |
0.00% |
31 |
73 |
Mozilla/5.0+(Linux;+Android+6.0;+CUBOT+DI NOSAUR+Build/ MRA58K)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 58.0.3029.83+Mobile+Safari/ 537.36 |
817 |
0.00% |
18 |
74 |
Mozilla/5.0+(compatible;+SeznamBot/ 3.2-test1;++http:/ / napoveda.seznam.cz/ en/ seznambot-intro/ ) |
769 |
0.00% |
258 |
75 |
Mozilla/5.0+(iPhone;+CPU+iPhone+OS+8_1+li ke+Mac+OS+X)+AppleWebKit/ 600.1.4+(KHTML,+like+Gecko)+Version/ 8.0+Mobile/ 12B411+Safari/ 600.1.4+(compatible;+YandexMobileBot/ 3.0;++http:/ / yandex.com/ bots) |
738 |
0.00% |
270 |
76 |
Mozilla/5.0+(compatible;+SeznamBot/ 3.2-test1-1;++http:/ / napoveda.seznam.cz/ en/ seznambot-intro/ ) |
681 |
0.00% |
219 |
77 |
Mozilla/5.0+(compatible;+Gluten+Free+Craw ler/ 1.0;++http:/ / glutenfreepleasure.com/ ) |
656 |
0.00% |
171 |
78 |
Mozilla/5.0+(compatible;+linkdexbot/ 2.2;++http:/ / www.linkdex.com/ bots/ ) |
645 |
0.00% |
483 |
79 |
Mozilla/5.0+(compatible;+YandexImageResiz er/ 2.0;++http:/ / yandex.com/ bots) |
607 |
0.00% |
572 |
80 |
Mozilla/5.0+(iPhone;+CPU+iPhone+OS+8_0+li ke+Mac+OS+X)+AppleWebKit/ 600.1.3+(KHTML,+like+Gecko)+Version/ 8.0+Mobile/ 12A4345d+Safari/ 600.1.4+moatbot |
596 |
0.00% |
52 |
81 |
Mozilla/5.0+(Linux;+Android+5.1;+CUBOT_NO TE_S+Build/ LMY47I)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 58.0.3029.83+Mobile+Safari/ 537.36 |
548 |
0.00% |
12 |
82 |
Mozilla/5.0+(compatible;+SMTBot/ 1.0;++http:/ / www.similartech.com/ smtbot) |
458 |
0.00% |
3 |
83 |
HubSpot+Links+Crawler+2.0+http:/ / www.hubspot.com/ |
391 |
0.00% |
151 |
84 |
SEMrushBot |
363 |
0.00% |
238 |
85 |
Gigabot Spider |
363 |
0.00% |
34 |
86 |
Mozilla/5.0+(Linux;+Android+6.0;+CUBOT+CH EETAH+2+Build/ MRA58K)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 58.0.3029.83+Mobile+Safari/ 537.36 |
350 |
0.00% |
5 |
87 |
DomainCrawler/3.0+(info@domaincrawler.com ;+http:/ / www.domaincrawler.com/ curezone.org) |
342 |
0.00% |
1 |
88 |
Mozilla/5.0+(compatible;+FemtosearchBot/ 1.0;+http:/ / femtosearch.com) |
340 |
0.00% |
220 |
89 |
RankFlex.com+Webspider |
330 |
0.00% |
102 |
90 |
Slack-ImgProxy+(+https:/ / api.slack.com/ robots) |
320 |
0.00% |
291 |
91 |
Mozilla/5.0+(Linux;+Android+6.0;+CUBOT+MA X+Build/ MRA58K)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 58.0.3029.83+Mobile+Safari/ 537.36 |
317 |
0.00% |
5 |
92 |
uipbot/1.0+(uipbot@semasio.net) |
294 |
0.00% |
6 |
93 |
Mediatoolkitbot+(complaints@mediatoolkit. com) |
291 |
0.00% |
147 |
94 |
Mozilla/5.0+(X11;+U;+Linux+Core+i7-4980HQ ;+de;+rv:32.0;+compatible;+JobboerseBot;+ http:/ / www.jobboerse.com/ bot.htm)+Gecko/ 20100101+Firefox/ 38.0 |
281 |
0.00% |
1 |
95 |
Mozilla/5.0+(Linux;+Android+6.0;+CUBOT+DI NOSAUR+Build/ MRA58K)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 59.0.3071.36+Mobile+Safari/ 537.36 |
271 |
0.00% |
4 |
96 |
PDFDriveCrawler+2.36+(+http:/ / www.pdfdrive.net+) |
270 |
0.00% |
89 |
97 |
Mozilla/5.0+(compatible;+changedetection/ 7.0;++http:/ / www.changedetection.com/ bot.html) |
265 |
0.00% |
53 |
98 |
W3C HTML-Code Validator |
256 |
0.00% |
226 |
99 |
Mozilla/5.0+(Linux;+Android+6.0;+CUBOT+DI NOSAUR+Build/ MRA58K;+wv)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Version/ 4.0+Chrome/ 44.0.2403.119+Mobile+Safari/ 537.36 |
244 |
0.00% |
6 |
100 |
LexxeBot/1.0+(lexxebot@lexxe.com) |
229 |
0.00% |
74 |
101 |
Mozilla/5.0+(compatible;+SeznamBot/ 3.2-test4;++http:/ / napoveda.seznam.cz/ en/ seznambot-intro/ ) |
226 |
0.00% |
85 |
102 |
Mozilla/5.0+(compatible;+Dow+Jones+Search bot) |
202 |
0.00% |
2 |
103 |
Mozilla/5.0+(compatible;+Go-http-client/ 1.1;++centurybot9@gmail.com) |
200 |
0.00% |
9 |
104 |
Blackboard+Safeassign/ 0.1+(a+Storm-based+Blackboard+Safeassign+ web-crawler;+https:/ / github.com/ DigitalPebble/ storm-crawler;+stormcrawler@digitalpebble .com) |
198 |
0.00% |
1 |
105 |
Googlebot-Video/1.0 |
183 |
0.00% |
1 |
106 |
Synthesio+Crawler+release+MonaLisa+(conta ct+at+synthesio+dot+fr) |
182 |
0.00% |
30 |
107 |
Screaming+Frog+SEO+Spider/ 4.1 |
176 |
0.00% |
48 |
108 |
Auto+Spider+1.0 |
170 |
0.00% |
16 |
109 |
Mozilla/5.0+(Linux;+Android+6.0;+CUBOT_NO TE_S+Build/ MRA58K)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 57.0.2987.132+Mobile+Safari/ 537.36 |
163 |
0.00% |
1 |
110 |
AdsBot-Google+(+http:/ / www.google.com/ adsbot.html) |
162 |
0.00% |
2 |
111 |
Mozilla/5.0+(Linux;+Android+6.0;+CUBOT+DI NOSAUR+Build/ MRA58K;+wv)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Version/ 4.0+Chrome/ 58.0.3029.83+Mobile+Safari/ 537.36 |
161 |
0.00% |
3 |
112 |
Mozilla/4.0+(compatible;+MSIE+8.0;+Window s+NT+6.0+;+Claritybot) |
158 |
0.00% |
154 |
113 |
Mozilla/5.0+(compatible;+SISTRIX+Crawler; +http:/ / crawler.sistrix.net/ ) |
149 |
0.00% |
110 |
114 |
Mozilla/5.0+(Linux;+U;+Android+4.2.2;+es- es;+S308+Build/ CUBOT)+AppleWebKit/ 534.30+(KHTML,+like+Gecko)+Version/ 4.0+Mobile+Safari/ 534.30 |
141 |
0.00% |
1 |
115 |
Mozilla/5.0+(compatible;+SemrushBot-SA/ 0.97;++http:/ / www.semrush.com/ bot.html) |
139 |
0.00% |
58 |
116 |
R6_CommentReader(www.radian6.com/ crawler) |
130 |
0.00% |
45 |
117 |
Mozilla/5.0+(iPhone;+CPU+iPhone+OS+8_1+li ke+Mac+OS+X)+AppleWebKit/ 600.1.4+(KHTML,+like+Gecko)+Version/ 8.0+Mobile/ 12B410+Safari/ 600.1.4+(Applebot/ 0.1;++http:/ / www.apple.com/ go/ applebot) |
127 |
0.00% |
40 |
118 |
Mozilla/5.0+(compatible;+NetSeer+crawler/ 2.0;++http:/ / www.netseer.com/ crawler.html;+crawler@netseer.com) |
124 |
0.00% |
103 |
119 |
Mozilla/5.0+(Linux;+Android+5.1;+CUBOT_NO TE_S+Build/ LMY47I)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 55.0.2883.91+Mobile+Safari/ 537.36+OPR/ 42.7.2246.114996 |
122 |
0.00% |
2 |
120 |
yacybot+(/global;+amd64+Linux+3.2.41-042s tab120.16;+java+1.7.0_121;+Europe/ en)+http:/ / yacy.net/ bot.html |
122 |
0.00% |
2 |
121 |
Mozilla/5.0+(TweetmemeBot/ 4.0;++http:/ / datasift.com/ bot.html)+Gecko/ 20100101+Firefox/ 31.0 |
121 |
0.00% |
51 |
122 |
WikiDo/1.1+(http://wikido.com;+crawler@wi kido.com) |
117 |
0.00% |
20 |
123 |
Googlebot |
115 |
0.00% |
81 |
124 |
Mozilla/5.0+(compatible;+archive.org_bot+ +http:/ / www.archive.org/ details/ archive.org_bot) |
109 |
0.00% |
33 |
125 |
Mozilla/5.0+(compatible;+StorygizeBot;+ht tp:/ / www.storygize.com) |
108 |
0.00% |
84 |
126 |
Mozilla/5.0+(compatible;+archive.org_bot; +Wayback+Machine+Live+Record;++http:/ / archive.org/ details/ archive.org_bot) |
105 |
0.00% |
94 |
127 |
Mozilla/5.0+(Linux;+Android+4.2.2;+CUBOT+ X6+Build/ JDQ39)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 58.0.3029.83+Mobile+Safari/ 537.36 |
104 |
0.00% |
2 |
128 |
Mozilla/5.0+(compatible;+seoscanners.net/ 1;++spider@seoscanners.net) |
104 |
0.00% |
39 |
129 |
yacybot+(-global;+amd64+Linux+3.16.0-4-am d64;+java+1.7.0_121;+Europe/ de)+http:/ / yacy.net/ bot.html |
102 |
0.00% |
20 |
130 |
Mozilla/5.0+(Linux;+Android+5.1;+CUBOT_NO TE_S+Build/ LMY47I)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 52.0.2743.98+Mobile+Safari/ 537.36 |
99 |
0.00% |
1 |
131 |
Mozilla/5.0+(Linux;+Android+6.0;+CUBOT_MA NITO+Build/ MRA58K)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 58.0.3029.83+Mobile+Safari/ 537.36 |
92 |
0.00% |
3 |
132 |
bitlybot/3.0+(+http:/ / bit.ly/ ) |
90 |
0.00% |
25 |
133 |
SafeSearch+microdata+crawler+(https:/ / safesearch.avira.com,+safesearch-abuse@av ira.com) |
89 |
0.00% |
39 |
134 |
Mozilla/5.0+(compatible;+SurdotlyBot/ 1.0;++http:/ / sur.ly/ bot.html) |
88 |
0.00% |
33 |
135 |
cis455crawler |
83 |
0.00% |
9 |
136 |
HybridBot+(hybrid.ru/ about.+If+our+bot+caused+problems+please+ contact+us.+Contact+email:+m.lyashkov@tar getix.net) |
81 |
0.00% |
54 |
137 |
Google Feedfetcher |
79 |
0.00% |
47 |
138 |
Speedy Spider |
76 |
0.00% |
2 |
139 |
Mozilla/5.0+(Linux;+Android+6.0;+CUBOT_MA NITO+Build/ MRA58K;+wv)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Version/ 4.0+Chrome/ 58.0.3029.83+Mobile+Safari/ 537.36 |
71 |
0.00% |
1 |
140 |
Mozilla/5.0+(Linux;+Android+6.0;+CUBOT+DI NOSAUR+Build/ MRA58K)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 55.0.2883.91+Mobile+Safari/ 537.36 |
70 |
0.00% |
2 |
141 |
Mozilla/5.0+(compatible;+PaperLiBot/ 2.1;+http:/ / support.paper.li/ entries/ 20023257-what-is-paper-li) |
70 |
0.00% |
60 |
142 |
Mozilla/5.0+(compatible;+heritrix/ 3.3.0-SNAPSHOT-20140702-2247++http:/ / archive.org/ details/ archive.org_bot) |
68 |
0.00% |
8 |
143 |
ArchiveTeam+ArchiveBot/ 20170510.02+(wpull+2.0.2)+and+not+Mozilla / 5.0+(Windows+NT+6.1;+WOW64)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 42.0.2311.90+Safari/ 537.36 |
68 |
0.00% |
6 |
144 |
Mozilla/5.0+(compatible;+Discordbot/ 2.0;++https:/ / discordapp.com) |
68 |
0.00% |
61 |
145 |
wheatt.com+bot+1.0,+Just+getting+some+met adata+for+users |
67 |
0.00% |
10 |
146 |
Mozilla/5.0+(Linux;+Android+5.1;+CUBOT_NO TE_S+Build/ LMY47I)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Version/ 4.0+Chrome/ 39.0.0.0+Mobile+Safari/ 537.36 |
65 |
0.00% |
3 |
147 |
LinqiaCrawlerBot/1.0+(eng@linqia.com) |
64 |
0.00% |
26 |
148 |
Mozilla/5.0+(compatible;+PiplBot;++http:/ / www.pipl.com/ bot/ ) |
64 |
0.00% |
32 |
149 |
Mozilla/5.0+(Windows+NT+6.1;+WOW64)+Apple WebKit/ 534++(KHTML,+like+Gecko)+MsnBot-Media+/ 1.0b |
64 |
0.00% |
8 |
150 |
Mozilla/5.0+(compatible)+SemanticScholarB ot+(+https:/ / www.semanticscholar.org/ crawler) |
64 |
0.00% |
29 |
151 |
Mozilla/5.0+(compatible;+Crawler/ 0.9;+http:/ / linkfluence.net/ ) |
63 |
0.00% |
2 |
152 |
Mozilla/5.0+(compatible;+Linkbot/ 1.0;++http:/ / ir.ee/ linkbot) |
60 |
0.00% |
2 |
153 |
DoCoMo Spider |
60 |
0.00% |
19 |
154 |
Mozilla/5.0+(compatible;+SemrushBot-SI/ 0.97;++http:/ / www.semrush.com/ bot.html) |
55 |
0.00% |
5 |
155 |
betaBot |
55 |
0.00% |
4 |
156 |
DomainStatsBot/1.0+(http:/ / domainstats.io/ our-bot) |
55 |
0.00% |
24 |
157 |
Screaming+Frog+SEO+Spider/ 7.2 |
54 |
0.00% |
21 |
158 |
Mozilla/5.0+(Linux;+Android+5.1;+CUBOT_NO TE_S+Build/ LMY47I)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 57.0.2987.132+Mobile+Safari/ 537.36 |
54 |
0.00% |
1 |
159 |
Mozilla/5.0+(Linux;+Android+6.0;+CUBOT+EC HO+Build/ MRA58K)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 58.0.3029.83+Mobile+Safari/ 537.36 |
53 |
0.00% |
1 |
160 |
Mozilla/5.0+(Linux;+Android+6.0;+CUBOT_NO TE_S+Build/ MRA58K)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 46.0.2490.76+Mobile+Safari/ 537.36 |
52 |
0.00% |
2 |
161 |
Mozilla/5.0+(Linux;+U;+Android+6.0;+CUBOT +CHEETAH+2+Build/ MRA58K;+wv)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Version/ 4.0+Chrome/ 58.0.3029.83+Mobile+Safari/ 537.36+OPR/ 25.0.2254.116879 |
52 |
0.00% |
1 |
162 |
Mozilla/5.0+(compatible;+special_archiver / 3.1.1++http:/ / www.archive.org/ details/ archive.org_bot) |
50 |
0.00% |
28 |
163 |
Mozilla/5.0+(compatible;+Uptimebot/ 1.0;++http:/ / www.uptime.com/ uptimebot) |
49 |
0.00% |
48 |
164 |
Mozilla/5.0+(Linux;+Android+4.2.2;+CUBOT+ X6+Build/ JCampano)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 58.0.3029.83+Mobile+Safari/ 537.36 |
49 |
0.00% |
1 |
165 |
Mozilla/5.0+(compatible;+oBot/ 2.3.1;+http:/ / filterdb.iss.net/ crawler/ ) |
49 |
0.00% |
9 |
166 |
Mozilla/5.0+(Linux;+U;+Android+6.0;+CUBOT +CHEETAH+2+Build/ MRA58K;+wv)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Version/ 4.0+Chrome/ 58.0.3029.83+Mobile+Safari/ 537.36+OPR/ 24.0.2254.115784 |
46 |
0.00% |
1 |
167 |
BaiduSpider |
45 |
0.00% |
5 |
168 |
studyres_com+download+bot,+see+http:/ / studyres.com/ public/ bot+for+additional+info |
44 |
0.00% |
20 |
169 |
Java+1.7+Apache+HttpClient+(Linux+x86_64) +/ +GnowitNewsbot+/ +Contact+information+at+http:/ / www.gnowit.com |
44 |
0.00% |
44 |
170 |
LinkedInBot/1.0+(compatible;+Mozilla/ 5.0;+Jakarta+Commons-HttpClient/ 3.1++http:/ / www.linkedin.com) |
43 |
0.00% |
18 |
171 |
PSbot Spider |
43 |
0.00% |
4 |
172 |
Mozilla/5.0+(compatible;+redditbot/ 1.0;++http:/ / www.reddit.com/ feedback) |
40 |
0.00% |
5 |
173 |
Intersearch.de Spider |
38 |
0.00% |
30 |
174 |
Slackbot-LinkExpanding+1.0+(+https:/ / api.slack.com/ robots) |
38 |
0.00% |
37 |
175 |
Alexa crawler |
36 |
0.00% |
20 |
176 |
Mozilla/5.0+(compatible;+YandexAccessibil ityBot/ 3.0;++http:/ / yandex.com/ bots) |
33 |
0.00% |
4 |
177 |
NinjaBot/2.0+(+http:/ / www.imninjas.com) |
33 |
0.00% |
33 |
178 |
BOT/0.1+(BOT+for+JCE) |
31 |
0.00% |
8 |
179 |
Elmer,+the+Thinglink+ImageBot+(http:/ / www.thinglink.com/ help/ ThinglinkImageBot) |
31 |
0.00% |
14 |
180 |
Virusdie+crawler/3.0 |
29 |
0.00% |
2 |
181 |
Mozilla/5.0+(Windows;+U;+Windows+NT+5.1;+ en-US)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 27.0.1453.116+Safari/ 537.36+HubSpot+Webcrawler+-+web-crawlers@ hubspot.com |
29 |
0.00% |
24 |
182 |
calculon+spider/0.1.0+mailto:abuse@yieldb ot.com |
29 |
0.00% |
29 |
183 |
Mozilla/5.0+(compatible;+Facebot+1.0;+htt ps:/ / developers.facebook.com/ docs/ sharing/ webmasters/ crawler) |
28 |
0.00% |
26 |
184 |
spiderman |
28 |
0.00% |
28 |
185 |
Mozilla/5.0+(compatible;+HackerfallBot;++ http:/ / hackerfall.com/ help/ bot) |
28 |
0.00% |
7 |
186 |
Mozilla/5.0+(compatible;+SpiderLing+(a+SP IDER+for+LINGustic+research);++http:/ / nlp.fi.muni.cz/ projects/ biwec/ ) |
26 |
0.00% |
8 |
187 |
Mozilla/5.0+(Linux;+U;+Android+4.2.2;+es- es;+S308+Build/ CUBOT)+AppleWebKit/ 534.30+(KHTML,+like+Gecko)+Version/ 4.0+Mobile+Safari/ 534.30+GSA/ 7.2.26.16.arm |
26 |
0.00% |
1 |
188 |
GumGum+Bot+(http://gumgum.com;+support@gu mgum.com) |
26 |
0.00% |
13 |
189 |
Mozilla/5.0+(compatible;+YandexMetrika/ 2.0;++http:/ / yandex.com/ bots) |
26 |
0.00% |
4 |
190 |
yacybot+(/global;+amd64+Windows+8+6.2;+ja va+1.8.0_25;+Europe/ fr)+http:/ / yacy.net/ bot.html |
24 |
0.00% |
12 |
191 |
SafeDNSBot+(https://www.safedns.com/ searchbot) |
23 |
0.00% |
5 |
192 |
ArchiveTeam+ArchiveBot/ 20170106.02+(wpull+2.0.2)+and+not+Mozilla / 5.0+(Windows+NT+6.1;+WOW64)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 42.0.2311.90+Safari/ 537.36 |
21 |
0.00% |
2 |
193 |
yacybot+(/global;+amd64+Linux+3.2.0-2-amd 64;+java+1.8.0_131;+America/ en)+http:/ / yacy.net/ bot.html |
20 |
0.00% |
2 |
194 |
Mozilla/5.0+(Linux;+Android+6.0;+CUBOT+DI NOSAUR+Build/ MRA58K)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 57.0.2987.132+Mobile+Safari/ 537.36 |
18 |
0.00% |
4 |
195 |
Mozilla/5.0+(compatible;+MJ12bot/ v1.4.7;+http:/ / www.majestic12.co.uk/ bot.php? +)/ Nutch-1.13 |
18 |
0.00% |
4 |
196 |
Screaming+Frog+SEO+Spider/ 7.1 |
17 |
0.00% |
7 |
197 |
Mozilla/5.0+(compatible;+MJ12bot/ v1.4.7;+http:/ / www.majestic12.co.uk/ bot.php? +) |
17 |
0.00% |
6 |
198 |
RockhallFundingCrawler+(+http:/ / www.rockhallfunding.com) |
16 |
0.00% |
6 |
199 |
yacybot+(/global;+amd64+Linux+4.9.16-gent oo;+java+1.8.0_121;+Europe/ de)+http:/ / yacy.net/ bot.html |
16 |
0.00% |
3 |
200 |
Phantom.js+bot |
16 |
0.00% |
2 |
|
Subtotal |
27,719,722 |
100.00% |
1,048,915 |
|
Total |
27,720,393 |
100.00% |
1,049,262 |
