Robots on a website: who visits and why

There are already more robots on the internet than people. Imperva's report for 2025 put automated traffic at 53 percent of all requests online, and people at 47. Of that 53, a full 40 went to harmful robots: password scanners, fraud and data theft. The well-behaved bots, including search engines, were the remaining 13. Cloudflare, counting HTML requests in June 2026, put robots at 57 percent against 43 percent people. A person may open five sites before a purchase; an AI agent may open thousands.

On Cloud9Cams the picture is similar. In the day to the morning of 5 November the servers built about 4 million pages. About half of them were requested by robots that name themselves honestly: search engines, SEO services, AI companies and social networks. More traffic came from programs without a clear name, and among those are the ones that look for passwords on the site.

A robot, or crawler, is a program that opens pages on its own, reads them and follows the links. With every request it sends a user-agent string — a short signature of who arrived. A browser signs as Chrome or Safari; Google's search robot signs as Googlebot. The site puts the rules for robots in a robots.txt file at the root of the domain. Below is who visits us, where the results of their crawl show up, whom to let in, and how to tell the rest not to.

How many robots and who they are

We count the requests for which the server rebuilt a page. Copies served from cache are not included, so the real robot flow is larger. The figures are for the last day: each known robot is counted on its own.

The noisiest guest is not a search engine. Semrush alone, an SEO tool, asked for about 530 thousand pages in a day. Google in the same window asked for about 34 thousand — fifteen times fewer. The search engine you usually build a site for walks quietly; the loudest guest is the one that gathers data for someone else's reports.

Search robots: the ones that bring people

A search robot crawls the site so a page can appear in the results. These are the only guests after which people arrive directly: someone types a query, sees the page and clicks. That is why they are not rate-limited and not closed.

Googlebot visits us about 34 thousand times a day, mostly looking like a phone: the user-agent describes a Nexus 5X and ends with "compatible; Googlebot/2.1". The real Googlebot comes from addresses such as 66.249.73.6 and 66.249.64.227. The result of its crawl is the Google index, and you can see which pages it took and which it dropped in Google Search Console. Google does not read a Crawl-delay line: it picks its own speed and slows down when the site starts answering with errors.

Bingbot walks six times more often than Google — about 220 thousand pages a day from addresses 40.77.167.48 and 52.167.144.206. The Bing index is not only for Bing itself: DuckDuckGo, Yahoo and Copilot answers lean on it heavily. Reports live in Bing Webmaster Tools.

The surprise leader among search engines is Amazon. Its Amzn-SearchBot asked for about 450 thousand pages a day from Amazon cloud addresses such as 98.94.225.199. Amazon says this robot collects pages for search in its products, for example Alexa, and does not use them to train AI. If robots.txt does not mention it, it follows the rules the site gave other search engines. It does not support Crawl-delay.

Applebot — about 84 thousand pages a day. It fills Siri, Spotlight and Safari suggestions, and the real Applebot comes from Apple's own network, from addresses such as 17.166.22.101. Yandex takes about 14 thousand pages from addresses 213.180.203.29 and 5.255.231.15; its reports are in Yandex Webmaster. Baiduspider from China — about nine and a half thousand. There is also a newcomer: Epochal-Search took about 26 thousand model-statistics pages in half a day from cloud addresses such as 137.131.13.234.

SEO robots: the noisiest guest

SEO robots build a map of the internet for paid services: who links to whom, which pages a site has, which queries it ranks for. The results of their crawl do not appear in search; they appear in the Semrush, Ahrefs, Moz and Majestic cabinets. Any subscriber can see them there, including your competitor who wants to know where your links come from.

Semrush signs as "SemrushBot/7~bl", where bl means backlinks. Our robots.txt asks it to pause 10 seconds between requests, and Semrush says it will honour that pause. But the pause is counted per domain, and the site has dozens of language subdomains. In half a day Semrush hit 61 of them from 40 addresses, for example 185.191.171.11 and 85.208.96.210. Each subdomain is a separate site with its own 10 seconds — that is where half a million pages a day come from.

AhrefsBot — about 13 thousand pages a day from addresses such as 51.89.71.80. Ahrefs feeds that data into its link database and its Yep search engine. Majestic's MJ12bot — about 11 thousand: its crawl is spread across volunteers' computers, so the addresses come from many networks. Moz's DotBot — about 9 thousand from 216.244.66.241; Domain Authority is built from that crawl. DataForSeoBot — about 48 thousand from servers 136.243.228.178 and neighbours. That service sells data through an API to other tools, so you will probably never see your site in a report of its own.

Visitors get nothing from SEO robots, but you do if you use those services or watch competitors in them. If you do not — close them with two lines: "User-agent: SemrushBot" and "Disallow: /". We chose the middle: SEO robots may walk, but with a pause.

AI robots: three different jobs

"AI robot" hides three different jobs, and each one needs its own decision.

The first collect text to train models. The most active here is Meta-ExternalAgent: about 340 thousand pages a day from Meta's network, from IPv6 addresses such as 2a03:2880:f814:4e::. Meta says this robot collects pages to train models and to index its products. Anthropic's ClaudeBot takes about 66 thousand pages from addresses 216.73.216.224 and neighbours, honours Crawl-delay, and we ask it for 5 seconds. Reflectionbot, which trains Reflection AI's open models, — about 16 thousand. Alibaba's TokenFoundryBot — about 12 thousand, ByteDance's Bytespider — about 11 thousand. CCBot builds the open Common Crawl archive anyone can download, and many models were trained on it. The result of training is nowhere to see: the page becomes part of the model's knowledge, and no link back to the site appears afterwards.

GPTBot, OpenAI's training robot, is fully closed here. First we asked it to wait a day between requests; that did not help, and the "Disallow: /" line it does honour. Two big search engines have separate flags: Google-Extended and Applebot-Extended. These are not robots, they are switches. Closing them in robots.txt keeps you in Google and Apple search while asking them not to train their models on your pages.

The second are search engines for AI. They build an index from which a chat or an AI agent takes an answer and puts a link to the source. That is OAI-SearchBot for ChatGPT search, Claude-SearchBot, PerplexityBot. The most active here are KeenableBot, about 45 thousand pages a day from cloud addresses such as 104.197.111.254, and ExaSearchBot, about 42 thousand. Keenable builds search for AI agents, and their answers cite the source page. Exa signs every request with a cryptographic signature, so you can verify it. Those robots give a direct benefit: a link from a chat answer brings a person.

The third arrive at a person's request. When someone asks ChatGPT, Claude or Perplexity to open a page, ChatGPT-User, Claude-User and Perplexity-User follow. That is not a site crawl; it is one visit for one reader. Some companies say these requests may not follow robots.txt, because a person started them.

Link previews

When a link to the site is sent in Facebook, Instagram or Messenger, facebookexternalhit follows — about 21 thousand times a day, from addresses such as 2a03:2880:f80c:23::. It takes the title, description and picture that make the link card. Previews in Telegram, X and other messengers work the same way. If you close that robot, the link arrives as a bare address with no picture and no title, so it is better to let it in.

Impostors and scanners

user-agent is just text, and you can write anything in it. In half a day the address 34.152.1.220 from Google's cloud introduced itself to us under thirty names: Googlebot, Bingbot, GPTBot, ClaudeBot, Amazonbot, CCBot, Baiduspider, GrokBot, DeepSeekBot and others, about a hundred and fifty times each. Most of the requests "from GPTBot" and "from Perplexity" in our logs came from it and from two neighbours, 34.23.74.162 and 35.231.45.174. The real GPTBot almost never appears here: it respects Disallow: /. What looks like OpenAI in the logs is mostly that impostor.

Others do the same. From the evening of 5 November to the morning of the 6th two addresses in Google's cloud, 35.254.143.178 and 34.126.114.217, made about 80 thousand requests and introduced themselves under thirty-three names: Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, GrokBot, Kimi-SearchBot, Qwenbot and others. Along the way they asked for subdomains the site does not have: guacamole, files, proxmox, kibana, jira, grafana.

The same address was doing something else: looking for files that should not be public on a normal site. In half a day one of our servers saw about 5 thousand such attempts. Most often they asked for /.env — the file with database passwords and service keys — and copies such as /.env.production and /.env.bak. Also /.git/config and /.git/HEAD, that is the code history, and /wp-login.php, the WordPress admin login, although we do not run WordPress. Those scanners most often sign as plain Chrome 131 or do not sign at all.

robots.txt will not stop them. Worse, a scanner reads robots.txt to find what the owner is hiding. So secret paths are not written into robots.txt: they are password-protected or not published at all.

A real robot can be checked by address. For Google the reverse record ends in googlebot.com or google.com, for Bing in search.msn.com, for Apple in applebot.apple.com, for Yandex in yandex.ru, yandex.net or yandex.com. Then that name is looked up again, and the address must match. Google, OpenAI, Amazon, Keenable and Reflection publish their address lists. An address that pretends to be a known robot and fails the check is refused here: instead of the page it gets an empty reply.

What it was like before robots.txt

Until mid-October 2026 we had almost no separate rules for other people's crawlers. Model profile pages held about 40–60 requests a second — one and a half to two million model pages a day on the European front alone, and almost none of that was people.

In the logs for the two hours before the ad launch on 13 October there were 460 thousand hits on profiles and only 5 thousand runs of the script that executes in a browser. The main stream disguised itself as Macintosh Chrome and did not name itself at all. Among the signed ones next to it were GPTBot and ClaudeBot. Later, after the first edits, GPT and Claude each held 15–17 requests a second on the chart — together more than two million pages a day from two training robots alone.

On 13 October we first asked GPT and Claude to come at most once a day. The pause did not help: OpenAI does not honour Crawl-delay. Then GPTBot, Google-Extended, Applebot-Extended and Amazonbot were closed completely with "Disallow: /". ClaudeBot was left with a 5-second pause, Semrush with 10, and the other non-search bots with 1 second. Amazonbot was slowed first: it ignored the pause, so it had to be closed. Now real GPTBot almost never appears, Claude walks about 66 thousand times a day instead of the earlier hundreds of thousands, and training Amazonbot is closed. We left Amzn-SearchBot for Amazon search alone.

What a webmaster should do

Write the rules at once, not when the server is already drowning. Many companies cache robots.txt for a long time: Amazon says it may keep a copy for up to 30 days; Perplexity and Reflection usually reread the file in about a day. While the robot walks with the old file, the new rule does nothing. So edit the file the same day a new guest shows up in the logs, not a week later "to see".

First open the server log and count which signatures appear most often. For the busiest ones check the address: a real Googlebot ends in googlebot.com, a real Bing in search.msn.com. An address that pretends to be a known robot and fails the check is not cured in robots.txt — it is cut on the server.

The robots.txt file sits at the site root and is made of groups. A group starts with a User-agent line naming the robot, then the rules. "User-agent: GPTBot" and "Disallow: /" close the whole site for GPTBot. "Disallow: /admin" closes only a section. Allow opens an exception inside a closed area. The "User-agent: *" group is for everyone not named. The robot picks one group — the one where its name is matched most precisely — and ignores the rest. If you opened a separate group for ClaudeBot with a pause, the bans from the common group no longer apply to it and must be repeated.

Crawl-delay is the pause in seconds between requests. Bing, Semrush, Ahrefs, ClaudeBot and Common Crawl honour it. Google and Amazon do not. The pause is counted for each domain and subdomain separately: if you have twenty language mirrors, twenty pauses run in parallel.

Whom to let in. Search robots — always: Googlebot, bingbot, YandexBot, Applebot, Baiduspider, DuckDuckBot. They are the only ones that bring people from the results. Link-preview robots too, or the card in a messenger stays bare. AI searchers such as OAI-SearchBot and visits at a person's request are useful if you want links from chat answers.

Whom to slow. SEO robots — SemrushBot, AhrefsBot, DotBot, MJ12bot, DataForSeoBot — if you use their cabinets yourself. A pause of 1–10 seconds is usually enough. If you do not use the cabinets, you can close them.

Whom to close. Training robots that send no links back: GPTBot, Amazonbot, Google-Extended, Applebot-Extended, Bytespider, CCBot, if you do not want to feed other people's models. PetalBot is closed here too. Impostors and password scanners are not stopped by robots.txt: they are cut by address.

robots.txt controls crawling, not a place in search. A closed page can still appear in the results with no description if other sites link to it. To remove a page from search you need a meta robots tag set to noindex or the same X-Robots-Tag response header, and the robot must be allowed to open the page or it will not see that tag. The last resort is the server reply: 429 or 503 for a while slows even Google.

Our own robots.txt is open: you can see whom we slowed and whom we closed. If the site already brings traffic and you are thinking how to earn on it, read our note on webcam affiliate programs.