|
Spider |
Hits |
% of Total |
Visits |
1 |
Bing Spider |
6,422,840 |
23.92% |
10,390 |
2 |
BaiDu Spider |
6,193,681 |
23.06% |
198,129 |
3 |
Google Spider |
3,750,451 |
13.97% |
41,569 |
4 |
Yandex Spider |
1,963,004 |
7.31% |
16,303 |
5 |
Wespe.de Spider |
1,515,948 |
5.64% |
793,469 |
6 |
Yahoo Spider |
1,344,342 |
5.01% |
2,870 |
7 |
magpie-crawler/1.1+(U;+Linux+amd64;+en-GB ;++http:/ / www.brandwatch.net) |
919,237 |
3.42% |
520 |
8 |
Treato-Bot/1.0+(TREATO+LTD,+www.treato.co m,+mailto:+botadmin@treato.com) |
716,568 |
2.67% |
105 |
9 |
Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10 _10_1)+AppleWebKit/ 600.2.5+(KHTML,+like+Gecko)+Version/ 8.0.2+Safari/ 600.2.5+(Applebot/ 0.1;++http:/ / www.apple.com/ go/ applebot) |
633,036 |
2.36% |
354 |
10 |
Vegi+bot+(we+follow+your+robots.txt+setti ngs+before+crawling,+you+can+slow+down+th e+bot+by+change+the+Crawl-Delay+parameter +in+the+settings.if+you+have+an+enquiry,+ please+email+to:+abuse-report@terrykylese oagency.com) |
591,544 |
2.20% |
29 |
11 |
Mozilla/5.0+(compatible;+MegaIndex.ru/ 2.0;++http:/ / megaindex.com/ crawler) |
443,313 |
1.65% |
17 |
12 |
MSN Spider |
272,680 |
1.02% |
628 |
13 |
Mozilla/5.0+(compatible;+BLEXBot/ 1.0;++http:/ / webmeup-crawler.com/ ) |
243,198 |
0.91% |
159 |
14 |
Sogou Spider |
231,699 |
0.86% |
9,171 |
15 |
Mozilla/5.0+(compatible;+DotBot/ 1.1;+http:/ / www.opensiteexplorer.org/ dotbot,+help@moz.com) |
211,479 |
0.79% |
5,984 |
16 |
Mozilla/5.0+(compatible;+MJ12bot/ v1.4.7;+http:/ / mj12bot.com/ ) |
202,744 |
0.75% |
1,269 |
17 |
Mozilla/5.0+(compatible;+SemrushBot/ 1.2~bl;++http:/ / www.semrush.com/ bot.html) |
182,282 |
0.68% |
2,624 |
18 |
Mozilla/5.0+(Windows;+U;+Windows+NT+6.0;+ en-GB;+rv:1.0;+trendictionbot0.5.0;+trend iction+search;+http:/ / www.trendiction.de/ bot;+please+let+us+know+of+any+problems;+ web+at+trendiction.com)+Gecko/ 20071127+Firefox/ 3.0.0.11 |
81,773 |
0.30% |
27,833 |
19 |
Facebook external hit |
76,062 |
0.28% |
52,960 |
20 |
Google image crawler |
75,559 |
0.28% |
640 |
21 |
Mozilla/5.0+(compatible;+proximic;++http: / / www.proximic.com/ info/ spider.php) |
69,937 |
0.26% |
61,357 |
22 |
Mozilla/5.0+(compatible;+SiteExplorer/ 1.1b;++http:/ / siteexplorer.info/ Backlink-Checker-Spider/ ) |
59,559 |
0.22% |
126 |
23 |
Mozilla/5.0+(compatible;+SeznamBot/ 3.2;++http:/ / napoveda.seznam.cz/ en/ seznambot-intro/ ) |
51,068 |
0.19% |
784 |
24 |
Mozilla/5.0+(compatible;+Gluten+Free+Craw ler/ 1.0;++http:/ / glutenfreepleasure.com/ ) |
47,382 |
0.18% |
193 |
25 |
Mozilla/5.0+(compatible;+Cliqzbot/ 1.0;++http:/ / cliqz.com/ company/ cliqzbot) |
46,766 |
0.17% |
1,886 |
26 |
R6_FeedFetcher_(www.radian6.com/ crawler) |
46,640 |
0.17% |
266 |
27 |
Clickagy+Intelligence+Bot+v2 |
45,590 |
0.17% |
30,884 |
28 |
Yandex Images Spider |
39,940 |
0.15% |
3,278 |
29 |
robots |
37,615 |
0.14% |
6,673 |
30 |
Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10 _10_1)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 40.0.2214.111+Safari/ 537.36+moatbot |
27,318 |
0.10% |
2,170 |
31 |
Mozilla/5.0+(compatible;+Steeler/ 3.5;+http:/ / www.tkl.iis.u-tokyo.ac.jp/ ~crawler/ ) |
26,943 |
0.10% |
835 |
32 |
istellabot/t.1 |
24,631 |
0.09% |
373 |
33 |
BoardReader/1.0+(http:/ / spider.boardreader.com) |
23,494 |
0.09% |
15,429 |
34 |
CCBot/2.0+(http://commoncrawl.org/ faq/ ) |
18,423 |
0.07% |
131 |
35 |
Mozilla/5.0+(compatible;+GrapeshotCrawler / 2.0;++http:/ / www.grapeshot.co.uk/ crawler.php) |
18,243 |
0.07% |
13,096 |
36 |
Mozilla/5.0+(compatible;+XoviBot/ 2.0;++http:/ / www.xovibot.net/ ) |
17,986 |
0.07% |
2,391 |
37 |
Mozilla/5.0+(compatible;+MJ12bot/ v1.4.8;+http:/ / mj12bot.com/ ) |
17,137 |
0.06% |
867 |
38 |
MaxPointCrawler/Nutch-1.10+(maxpoint.craw ler+at+maxpointinteractive+dot+com) |
13,886 |
0.05% |
860 |
39 |
Mozilla/5.0+(compatible;+ZumBot/ 1.0;+http:/ / help.zum.com/ inquiry) |
13,494 |
0.05% |
0 |
40 |
GarlikCrawler/1.2+(http:/ / garlik.com/ ,+crawler@garlik.com) |
11,238 |
0.04% |
258 |
41 |
Mozilla/5.0+(compatible;+MSIE+9.0;+Window s+NT+6.1;+Trident/ 5.0);+360Spider |
11,092 |
0.04% |
2,567 |
42 |
Pinterest/0.2+(+http:/ / www.pinterest.com/ bot.html) |
10,424 |
0.04% |
2,858 |
43 |
Mozilla/5.0+(compatible;+spbot/ 5.0.3;++http:/ / OpenLinkProfiler.org/ bot+) |
6,955 |
0.03% |
17 |
44 |
Mozilla/5.0+(compatible;+NTENTbot;++http: / / www.ntent.com/ ntentbot) |
6,472 |
0.02% |
60 |
45 |
Mozilla/5.0+(Windows+NT+6.2)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 31.0.1650.63+Safari/ 537.36+QIHU+360SE;+360Spider |
6,243 |
0.02% |
2,189 |
46 |
Mozilla/5.0+(Linux;+Android+6.0;+CUBOT_NO TE_S+Build/ MRA58K)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 56.0.2924.87+Mobile+Safari/ 537.36 |
5,262 |
0.02% |
66 |
47 |
IAS+crawler+(ias_crawler;+http:/ / integralads.com/ site-indexing-policy/ ) |
5,150 |
0.02% |
385 |
48 |
Google Feedfetcher |
4,599 |
0.02% |
2,050 |
49 |
Mozilla/5.0+(compatible;+linkdexbot/ 2.0;++http:/ / www.linkdex.com/ bots/ ) |
4,538 |
0.02% |
2,004 |
50 |
Mozilla/5.0+(compatible;+Linux+x86_64;+Ma il.RU_Bot/ 2.0;++http:/ / go.mail.ru/ help/ robots) |
4,306 |
0.02% |
301 |
51 |
YisouSpider |
4,131 |
0.02% |
2,116 |
52 |
Mozilla/5.0+(compatible;+Linux+x86_64;+Ma il.RU_Bot/ Img/ 2.0;++http:/ / go.mail.ru/ help/ robots) |
3,354 |
0.01% |
1,626 |
53 |
Blackboard+Safeassign/ 0.1+(a+Storm-based+Blackboard+Safeassign+ web-crawler;+https:/ / github.com/ DigitalPebble/ storm-crawler;+stormcrawler@digitalpebble .com) |
3,255 |
0.01% |
38 |
54 |
Domain+Re-Animator+Bot+(http:/ / domainreanimator.com)+-+support@domainrea nimator.com |
2,559 |
0.01% |
5 |
55 |
SEMrushBot |
2,319 |
0.01% |
181 |
56 |
Alexa Spider |
2,189 |
0.01% |
244 |
57 |
PDF+Drive+Crawler+1.02+(+http:/ / www.pdfdrive.net+) |
2,103 |
0.01% |
724 |
58 |
adbeat_bot |
2,062 |
0.01% |
777 |
59 |
GetIntent+Crawler+(http:/ / getintent.com/ bot.html) |
1,933 |
0.01% |
1,328 |
60 |
Twitterbot/1.0 |
1,771 |
0.01% |
808 |
61 |
Mozilla/5.0+(Windows;+U;+Windows+NT+5.1;+ en-US;+rv:1.8.1.6)+Gecko/ 20070725+Firefox/ 2.0.0.6+-+James+BOT+-+WebCrawler+http:/ / cognitiveseo.com/ bot.html |
1,674 |
0.01% |
587 |
62 |
Googlebot+(gocrawl+v0.4) |
1,626 |
0.01% |
767 |
63 |
Mozilla/5.0+(compatible;+MagiBot-Matarael / 5.0.0;++http:/ / www.peak-labs.com) |
1,373 |
0.01% |
87 |
64 |
Mozilla/5.0+(compatible;+MojeekBot/ 0.6;++https:/ / www.mojeek.com/ bot.html) |
1,340 |
0.00% |
580 |
65 |
Mozilla/5.0+(iPhone;+CPU+iPhone+OS+8_1+li ke+Mac+OS+X)+AppleWebKit/ 600.1.4+(KHTML,+like+Gecko)+Version/ 8.0+Mobile/ 12B411+Safari/ 600.1.4+(compatible;+YandexMobileBot/ 3.0;++http:/ / yandex.com/ bots) |
1,327 |
0.00% |
556 |
66 |
Mozilla/5.0+(+compatible+;+Veooz/ 1.0+;++http:/ / www.veooz.com/ veoozbot.html+) |
1,282 |
0.00% |
335 |
67 |
Mozilla/5.0+(compatible;+Yeti/ 1.1;++http:/ / naver.me/ bot) |
1,062 |
0.00% |
1,001 |
68 |
Mozilla/5.0+(compatible;+WBSearchBot/ 1.1;++http:/ / www.warebay.com/ bot.html) |
1,038 |
0.00% |
8 |
69 |
Mozilla/5.0+(compatible;+coccocbot-image/ 1.0;++http:/ / help.coccoc.com/ searchengine) |
1,025 |
0.00% |
251 |
70 |
Mozilla/5.0+(compatible;+coccocbot-web/ 1.0;++http:/ / help.coccoc.com/ searchengine) |
932 |
0.00% |
472 |
71 |
Mozilla/5.0+(Linux;+Android+6.0;+CUBOT+MA X+Build/ MRA58K)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 56.0.2924.87+Mobile+Safari/ 537.36 |
902 |
0.00% |
16 |
72 |
Mozilla/5.0+(Linux;+Android+6.0;+CUBOT+DI NOSAUR+Build/ MRA58K)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 56.0.2924.87+Mobile+Safari/ 537.36 |
883 |
0.00% |
25 |
73 |
Mozilla/5.0+(compatible;+linkdexbot/ 2.2;++http:/ / www.linkdex.com/ bots/ ) |
837 |
0.00% |
657 |
74 |
Mozilla/5.0+(compatible;+SeznamBot/ 3.2-test1;++http:/ / napoveda.seznam.cz/ en/ seznambot-intro/ ) |
790 |
0.00% |
270 |
75 |
Mozilla/5.0+(compatible;+SeznamBot/ 3.2-test1-1;++http:/ / napoveda.seznam.cz/ en/ seznambot-intro/ ) |
785 |
0.00% |
271 |
76 |
Mozilla/5.0+(X11;+U;+Linux+Core+i7-4980HQ ;+de;+rv:32.0;+compatible;+JobboerseBot;+ http:/ / www.jobboerse.com/ bot.htm)+Gecko/ 20100101+Firefox/ 38.0 |
636 |
0.00% |
2 |
77 |
Mozilla/5.0+(compatible;+SMTBot/ 1.0;++http:/ / www.similartech.com/ smtbot) |
631 |
0.00% |
4 |
78 |
TinEye-bot/0.61+(see+http:/ / www.tineye.com/ crawler.html) |
623 |
0.00% |
76 |
79 |
Aboundex/0.3+(http:// www.aboundex.com/ crawler/ ) |
600 |
0.00% |
2 |
80 |
Googlebot-Video/1.0 |
552 |
0.00% |
4 |
81 |
HubSpot+Links+Crawler+2.0+http:/ / www.hubspot.com/ |
527 |
0.00% |
182 |
82 |
PSbot Spider |
515 |
0.00% |
35 |
83 |
Synthesio+Crawler+release+MonaLisa+(conta ct+at+synthesio+dot+fr) |
478 |
0.00% |
11 |
84 |
Mozilla/5.0+(compatible;+YandexImageResiz er/ 2.0;++http:/ / yandex.com/ bots) |
465 |
0.00% |
452 |
85 |
Slack-ImgProxy+(+https:/ / api.slack.com/ robots) |
446 |
0.00% |
345 |
86 |
Mozilla/5.0+(compatible;+seoscanners.net/ 1;++spider@seoscanners.net) |
377 |
0.00% |
212 |
87 |
Mozilla/5.0+(Linux;+Android+5.1;+CUBOT_NO TE_S+Build/ LMY47I)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Version/ 4.0+Chrome/ 39.0.0.0+Mobile+Safari/ 537.36 |
365 |
0.00% |
9 |
88 |
Mozilla/5.0+(compatible)+SemanticScholarB ot+(+https:/ / www.semanticscholar.org/ crawler) |
351 |
0.00% |
49 |
89 |
RankFlex.com+Webspider |
342 |
0.00% |
114 |
90 |
Mozilla/5.0+(iPhone;+CPU+iPhone+OS+8_0+li ke+Mac+OS+X)+AppleWebKit/ 600.1.3+(KHTML,+like+Gecko)+Version/ 8.0+Mobile/ 12A4345d+Safari/ 600.1.4+moatbot |
330 |
0.00% |
28 |
91 |
Gigabot Spider |
315 |
0.00% |
21 |
92 |
Mozilla/5.0+(Linux;+Android+6.0;+CUBOT_NO TE_S+Build/ MRA58K;+wv)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Version/ 4.0+Chrome/ 49.0.2623.105+Mobile+Safari/ 537.36+ACHEETAHI/ 2100502066 |
296 |
0.00% |
6 |
93 |
Speedy Spider |
293 |
0.00% |
5 |
94 |
Mozilla/4.0+(compatible;+MSIE+6.0;+Window s+NT+5.1;+SV1;++http:/ / www.changedetection.com/ bot.html+) |
285 |
0.00% |
57 |
95 |
uipbot/1.0+(uipbot@semasio.net) |
274 |
0.00% |
10 |
96 |
Mozilla/5.0+(Linux;+Android+6.0.1;+Nexus+ 5X+Build/ MMB29P)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 41.0.2272.96+Mobile+Safari/ 537.36+(compatible;+Pinterestbot/ 1.0;++http:/ / www.pinterest.com/ bot.html) |
271 |
0.00% |
161 |
97 |
Mozilla/5.0+(compatible;+DuckDuckGo-Favic ons-Bot/ 1.0;++http:/ / duckduckgo.com) |
268 |
0.00% |
100 |
98 |
Mozilla/5.0+(compatible;+Cloudflare-AMP/ 1.0;++https:/ / amp.cloudflare.com/ bot)+AppleWebKit/ 534.34 |
263 |
0.00% |
257 |
99 |
Mozilla/5.0+(compatible;+oBot/ 2.3.1;+http:/ / filterdb.iss.net/ crawler/ ) |
259 |
0.00% |
4 |
100 |
Mozilla/5.0+(Linux;+Android+5.1;+CUBOT_NO TE_S+Build/ LMY47I)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 56.0.2924.87+Mobile+Safari/ 537.36 |
252 |
0.00% |
8 |
101 |
RSSingBot+(http://www.rssing.com) |
246 |
0.00% |
60 |
102 |
Mediatoolkitbot+(complaints@mediatoolkit. com) |
240 |
0.00% |
126 |
103 |
WikiDo/1.1+(http://wikido.com;+crawler@wi kido.com) |
229 |
0.00% |
41 |
104 |
BaiduSpider |
228 |
0.00% |
5 |
105 |
Mozilla/5.0+(compatible;+archive.org_bot+ +http:/ / www.archive.org/ details/ archive.org_bot) |
226 |
0.00% |
91 |
106 |
Mozilla/5.0+(compatible;+IDBot/ 1.1;++http:/ / www.id-search.xyz/ bot.html) |
224 |
0.00% |
8 |
107 |
Mozilla/5.0+(TweetmemeBot/ 4.0;++http:/ / datasift.com/ bot.html)+Gecko/ 20100101+Firefox/ 31.0 |
222 |
0.00% |
150 |
108 |
spider |
219 |
0.00% |
107 |
109 |
Mozilla/5.0+(compatible;+OrangeBot/ 2.0;+support.orangebot@orange.com) |
216 |
0.00% |
177 |
110 |
Mozilla/5.0+(compatible;+SeznamBot/ 3.2-test4;++http:/ / napoveda.seznam.cz/ en/ seznambot-intro/ ) |
210 |
0.00% |
80 |
111 |
Mozilla/5.0+(compatible;+archive.org_bot; +Wayback+Machine+Live+Record;++http:/ / archive.org/ details/ archive.org_bot) |
206 |
0.00% |
201 |
112 |
W3C HTML-Code Validator |
205 |
0.00% |
199 |
113 |
R6_CommentReader(www.radian6.com/ crawler) |
196 |
0.00% |
59 |
114 |
Mozilla/5.0+(compatible;+NetSeer+crawler/ 2.0;++http:/ / www.netseer.com/ crawler.html;+crawler@netseer.com) |
176 |
0.00% |
139 |
115 |
VeriCiteCrawler/Nutch-1.9 |
175 |
0.00% |
7 |
116 |
Mozilla/5.0+(compatible;+SemrushBot-SA/ 0.97;++http:/ / www.semrush.com/ bot.html) |
166 |
0.00% |
65 |
117 |
Screaming+Frog+SEO+Spider/ 7.1 |
156 |
0.00% |
79 |
118 |
Mozilla/5.0+(compatible;+SISTRIX+Crawler; +http:/ / crawler.sistrix.net/ ) |
154 |
0.00% |
109 |
119 |
ZoomBot+(Linkbot+1.0+http:/ / suite.seozoom.it/ bot.html) |
142 |
0.00% |
0 |
120 |
Mozilla/5.0+(compatible;+archive.org_bot/ 3.3.0++http:/ / pandora.nla.gov.au/ crawl.html) |
142 |
0.00% |
4 |
121 |
Mozilla/5.0+(Windows+NT+6.1;+WOW64)+Apple WebKit/ 534++(KHTML,+like+Gecko)+MsnBot-Media+/ 1.0b |
141 |
0.00% |
11 |
122 |
LinqiaCrawlerBot/1.0+(eng@linqia.com) |
132 |
0.00% |
54 |
123 |
ConBot |
132 |
0.00% |
126 |
124 |
DomainStatsBot/1.0+(http:/ / domainstats.io/ our-bot) |
131 |
0.00% |
42 |
125 |
Mozilla/5.0+(compatible;+Uptimebot/ 1.0;++http:/ / www.uptime.com/ uptimebot) |
128 |
0.00% |
126 |
126 |
ArchiveTeam+ArchiveBot/ 20170106.02+(wpull+2.0.2)+and+not+Mozilla / 5.0+(Windows+NT+6.1;+WOW64)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 42.0.2311.90+Safari/ 537.36 |
121 |
0.00% |
17 |
127 |
Mozilla/5.0+(compatible;+Linkbot/ 1.0;++http:/ / ir.ee/ linkbot) |
115 |
0.00% |
4 |
128 |
Mozilla/5.0+(compatible;+redditbot/ 1.0;++http:/ / www.reddit.com/ feedback) |
112 |
0.00% |
7 |
129 |
HELLO+Crawler |
106 |
0.00% |
21 |
130 |
Mozilla/5.0+(compatible;+PaperLiBot/ 2.1;+http:/ / support.paper.li/ entries/ 20023257-what-is-paper-li) |
101 |
0.00% |
96 |
131 |
Screaming+Frog+SEO+Spider/ 7.2 |
99 |
0.00% |
31 |
132 |
Mozilla/5.0+(Windows+NT+10.0;+ZumBot/ 1.0;+http:/ / help.zum.com/ ;WOW64;Trident/ 7.0;rv:11.0)+Chrome/ 50.0.2661.94+(KHTML,+like+Gecko) |
96 |
0.00% |
0 |
133 |
Mozilla/5.0+(Linux;+Android+5.1;+CUBOT_NO TE_S+Build/ LMY47I)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 55.0.2883.91+Mobile+Safari/ 537.36 |
93 |
0.00% |
2 |
134 |
yacybot+(/global;+amd64+FreeBSD+10.3-RELE ASE-p11;+java+1.8.0_112;+GMT/ en)+http:/ / yacy.net/ bot.html |
91 |
0.00% |
18 |
135 |
RSS-reader of Mac OS X |
88 |
0.00% |
44 |
136 |
Mozilla/5.0+(compatible;+PiplBot;++http:/ / www.pipl.com/ bot/ ) |
81 |
0.00% |
41 |
137 |
Mozilla/5.0+(compatible;+SemrushBot-SI/ 0.97;++http:/ / www.semrush.com/ bot.html) |
81 |
0.00% |
4 |
138 |
Mozilla/5.0+(iPhone;+CPU+iPhone+OS+8_1+li ke+Mac+OS+X)+AppleWebKit/ 600.1.4+(KHTML,+like+Gecko)+Version/ 8.0+Mobile/ 12B410+Safari/ 600.1.4+(Applebot/ 0.1;++http:/ / www.apple.com/ go/ applebot) |
80 |
0.00% |
18 |
139 |
DoCoMo Spider |
76 |
0.00% |
11 |
140 |
BOT/0.1+(BOT+for+JCE) |
75 |
0.00% |
9 |
141 |
Mozilla/5.0+(compatible;+YaK/ 1.0;+http:/ / linkfluence.com/ ;+bot@linkfluence.com) |
75 |
0.00% |
11 |
142 |
Mozilla/5.0+(compatible;+SurdotlyBot/ 1.0;++http:/ / sur.ly/ bot.html) |
74 |
0.00% |
33 |
143 |
Mozilla/5.0+(compatible;+special_archiver / 3.1.1++http:/ / www.archive.org/ details/ archive.org_bot) |
71 |
0.00% |
37 |
144 |
SafeSearch+microdata+crawler+(https:/ / safesearch.avira.com,+safesearch-abuse@av ira.com) |
69 |
0.00% |
62 |
145 |
Mozilla/5.0+(Linux;+U;+Android+4.2.2;+en- us;+CUBOT+P9+Build/ JDQ39)+AppleWebKit/ 534.30+(KHTML,+like+Gecko)+Version/ 4.0+Mobile+Safari/ 534.30 |
67 |
0.00% |
1 |
146 |
Mozilla/5.0+(Linux;+Android+6.0;+CUBOT_NO TE_S+Build/ MRA58K)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 57.0.2987.132+Mobile+Safari/ 537.36 |
64 |
0.00% |
1 |
147 |
Elmer,+the+Thinglink+ImageBot+(http:/ / www.thinglink.com/ help/ ThinglinkImageBot) |
64 |
0.00% |
34 |
148 |
Mozilla/5.0+(Linux;+Android+5.1;+CUBOT_NO TE_S+Build/ LMY47I)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Version/ 4.0+Chrome/ 39.0.0.0+Mobile+Safari/ 537.36+ACHEETAHI/ 1 |
62 |
0.00% |
1 |
149 |
Mozilla/5.0+(Linux;+Android+5.1;+CUBOT_NO TE_S+Build/ LMY47I)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 54.0.2840.68+Mobile+Safari/ 537.36 |
62 |
0.00% |
1 |
150 |
Screaming+Frog+SEO+Spider/ 4.1 |
61 |
0.00% |
19 |
151 |
Mozilla/5.0+(Linux;+U;+Android+4.2.2;+de- de;+CUBOT+P9+Build/ JDQ39)+AppleWebKit/ 534.30+(KHTML,+like+Gecko)+Version/ 4.0+Mobile+Safari/ 534.30 |
60 |
0.00% |
1 |
152 |
Mozilla/5.0+(Linux;+U;+Android+6.0;+en-US ;+CUBOT+DINOSAUR+Build/ MRA58K)+AppleWebKit/ 534.30+(KHTML,+like+Gecko)+Version/ 4.0+UCBrowser/ 11.2.0.915+U3/ 0.8.0+Mobile+Safari/ 534.30 |
58 |
0.00% |
1 |
153 |
Mozilla/5.0+(Linux;+Android+6.0;+CUBOT+DI NOSAUR+Build/ MRA58K;+wv)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Version/ 4.0+Chrome/ 44.0.2403.119+Mobile+Safari/ 537.36 |
58 |
0.00% |
1 |
154 |
Slackbot-LinkExpanding+1.0+(+https:/ / api.slack.com/ robots) |
57 |
0.00% |
54 |
155 |
Mozilla/5.0+(Windows+NT+6.3;compatible;+L eikibot/ 1.0;++http:/ / www.leiki.com) |
56 |
0.00% |
54 |
156 |
ArchiveTeam+ArchiveBot/ 20170106.02+(wpull+2.0.1)+and+not+Mozilla / 5.0+(Windows+NT+6.1;+WOW64)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 42.0.2311.90+Safari/ 537.36 |
56 |
0.00% |
9 |
157 |
Mozilla/5.0+(Windows+NT+10.0;+WOW64)+Appl eWebKit/ 537.36+(KHTML,+like+Gecko;+compatible;+Ma ilDBBot/ 1.0;++https:/ / maildb.io/ bot)+Safari/ 537.36 |
56 |
0.00% |
12 |
158 |
Mozilla/5.0+(compatible;+YandexAccessibil ityBot/ 3.0;++http:/ / yandex.com/ bots) |
55 |
0.00% |
14 |
159 |
Mozilla/5.0+(compatible;+Discordbot/ 1.0;++https:/ / discordapp.com) |
55 |
0.00% |
48 |
160 |
bitlybot/3.0+(+http:/ / bit.ly/ ) |
50 |
0.00% |
18 |
161 |
Mozilla/5.0+(Linux;+Android+4.2.1;+CUBOT+ GT99+Build/ JOP40D)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 56.0.2924.87+Mobile+Safari/ 537.36 |
49 |
0.00% |
1 |
162 |
Test+Spider/Nutch-1.12 |
46 |
0.00% |
11 |
163 |
Intersearch.de Spider |
44 |
0.00% |
38 |
164 |
HybridBot+(hybrid.ru/ about.+If+our+bot+caused+problems+please+ contact+us.+Contact+email:+m.lyashkov@tar getix.net) |
44 |
0.00% |
34 |
165 |
Mozilla/5.0+(Linux;+Android+6.0;+CUBOT+EC HO+Build/ MRA58K)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 56.0.2924.87+Mobile+Safari/ 537.36 |
44 |
0.00% |
2 |
166 |
Googlebot |
43 |
0.00% |
32 |
167 |
GumGum+Bot+(http://gumgum.com;+support@gu mgum.com) |
42 |
0.00% |
21 |
168 |
ssearch_bot+(sSearch+Crawler;+http:/ / www.semantissimo.de) |
41 |
0.00% |
29 |
169 |
Mozilla/5.0+(compatible;+heritrix/ 3.3.0++http:/ / netarkivet.dk/ webcrawler/ ) |
41 |
0.00% |
2 |
170 |
Mozilla/5.0+(Windows;+U;+Windows+NT+5.1;+ en;+rv:1.9.0.13)+Gecko/ 2009073022+Firefox/ 3.5.2+(.NET+CLR+3.5.30729)+SurveyBot/ 2.3+(DomainTools) |
41 |
0.00% |
27 |
171 |
Mozilla/5.0+(compatible;+HackerfallBot;++ http:/ / hackerfall.com/ help/ bot) |
36 |
0.00% |
9 |
172 |
MSIECrawler |
36 |
0.00% |
8 |
173 |
Mozilla/5.0+(compatible;+YesupBot/ 1.0;++http:/ / www.yesup.net/ bot.html) |
35 |
0.00% |
33 |
174 |
Mozilla/5.0+(Linux;+Android+6.0;+CUBOT_MA NITO+Build/ MRA58K)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 53.0.2785.124+Mobile+Safari/ 537.36 |
34 |
0.00% |
0 |
175 |
Mozilla/5.0+(compatible;+spbot/ 4.0.7;++https:/ / www.seoprofiler.com/ bot+) |
32 |
0.00% |
8 |
176 |
Auto+Spider+1.0 |
30 |
0.00% |
2 |
177 |
calculon+spider/0.1.0+mailto:abuse@yieldb ot.com |
30 |
0.00% |
30 |
178 |
Mozilla/5.0+(Linux;+Android+6.0;+CUBOT_NO TE_S+Build/ MRA58K)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 46.0.2490.76+Mobile+Safari/ 537.36 |
29 |
0.00% |
3 |
179 |
SafeDNSBot+(https://www.safedns.com/ searchbot) |
28 |
0.00% |
5 |
180 |
Mozilla/5.0+(compatible;+YandexMetrika/ 2.0;++http:/ / yandex.com/ bots) |
26 |
0.00% |
4 |
181 |
Mozilla/5.0+(compatible;+heritrix/ 3.3.0-SNAPSHOT-20140702-2247++http:/ / archive.org/ details/ archive.org_bot) |
26 |
0.00% |
2 |
182 |
Java+1.7+Apache+HttpClient+(Linux+x86_64) +/ +GnowitNewsbot+/ +Contact+information+at+http:/ / www.gnowit.com |
24 |
0.00% |
24 |
183 |
johnhew+crawler,+johnhew@seas.upenn.edu/ Nutch-1.12 |
24 |
0.00% |
12 |
184 |
Web+Fire+Spider/Nutch-1.12 |
23 |
0.00% |
8 |
185 |
Search4free Spider |
23 |
0.00% |
5 |
186 |
NinjaBot/2.0+(+http:/ / www.imninjas.com) |
22 |
0.00% |
22 |
187 |
My+Nutch+Spider/Nutch-1.12 |
21 |
0.00% |
7 |
188 |
Xaldon+WebSpider |
21 |
0.00% |
5 |
189 |
Feedjit+Favicon+Crawler+1.0 |
20 |
0.00% |
20 |
190 |
Mozilla/5.0+(Windows+NT+6.1;+rv:38.0)+Gec ko/ 20100101+Firefox/ 38.0+(IndeedBot+1.1) |
20 |
0.00% |
2 |
191 |
LinkedInBot/1.0+(compatible;+Mozilla/ 5.0;+Jakarta+Commons-HttpClient/ 3.1++http:/ / www.linkedin.com) |
20 |
0.00% |
9 |
192 |
Mozilla/5.0+(Linux;+Android+5.1;+CUBOT_NO TE_S+Build/ LMY47I)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 58.0.3029.21+Mobile+Safari/ 537.36 |
19 |
0.00% |
0 |
193 |
SocialRankIOBot;+http:/ / socialrank.io/ about |
19 |
0.00% |
17 |
194 |
Mozilla/5.0+(Windows+NT+6.2;+Win64;+x64)+ AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 32.0.1667.0+Safari/ 537.36+_zbot |
18 |
0.00% |
10 |
195 |
Mozilla/5.0+(Linux;+Android+6.0;+CUBOT+DI NOSAUR+Build/ MRA58K)+AppleWebKit/ 537.36+(KHTML,+like+Gecko)+Chrome/ 55.0.2883.91+Mobile+Safari/ 537.36 |
18 |
0.00% |
1 |
196 |
AppleNewsBot |
17 |
0.00% |
15 |
197 |
Mozilla/5.0+(compatible;+Windows+NT+6.1? ;+ZumBot/ 1.0;+http:/ / help.zum.com/ inquiry) |
17 |
0.00% |
6 |
198 |
Mozilla/5.0+(compatible;+SMTBot/ 1.0;+http:/ / www.similartech.com/ smtbot) |
16 |
0.00% |
4 |
199 |
Phantom.js+bot |
16 |
0.00% |
2 |
200 |
OpenTaggerBot+(http:/ / www.opentagger.com/ opentaggerbot.htm) |
16 |
0.00% |
0 |
|
Subtotal |
26,854,775 |
100.00% |
1,338,532 |
|
Total |
26,855,618 |
100.00% |
1,338,931 |
