To research this note, I asked Claude Code one question. It sent four research agents out to read about bots, and between them they made about 480 tool calls, mostly searches and page fetches, in a little over half an hour. Partway through, Cloudflare's blog started answering one of them with a bot challenge while it was reading Cloudflare's own posts about bots. I can't tell what triggered the challenge, and I'm not assuming it was a mistake.
That agent was a bot by any definition. It was also doing something I had asked for, which puts it somewhere between a crawler running on a company's schedule and me, and the word doesn't tell a website which of those it's looking at. Matthew Prince, the CEO of Cloudflare (where I was Field CTO), put it bluntly in June: "Bot / Crawler / Agent are all synonyms depending on whether you want them to be a good or bad thing, normatively." Cloudflare's own blog said in September that "'AI bot' no longer means anything useful."
What the word used to cover
For a long time "bot" meant two quite different things. Some bots read the web: search crawlers going back to the early 1990s, plus uptime monitors, link previews and feed readers. Sites asked crawlers to stay out of certain areas with robots.txt, a convention from 1994 that was always a polite request; the original document says it is "not enforced by anybody."
Other bots bought or broke things. In 2014 one ticket broker's bot bought 1,012 tickets to a U2 show at Madison Square Garden in a single minute, despite a four-ticket limit, according to the New York Attorney General. I learned how that feels from the other side when the PlayStation 5 was impossible to find. Retailers released them in batches, I usually lost to the bots, and a few times I'm pretty sure Walmart's cart crashed while I was trying. Walmart later said that one fix it put in just hours before its PlayStation 5 sale on Nov. 25, 2020 "blocked more than 20 million bot attempts within the first 30 minutes alone." Bots of that kind never went away. Thales, which measures its own customers' traffic, is a different count: it put bots at 53% of what it saw in 2025, and bad bots alone at 40%.
Taylor Swift's Eras Tour presale in 2022 is the case most people remember. Bot traffic helped overwhelm the presale, and Ticketmaster had to slow it down and then pause it, but Live Nation's president testified that the bots didn't get tickets. The FTC later sued a Maryland broker, alleging it used 49 accounts to buy 273 tickets to a single Eras Tour show (the broker denies using bots), and in April 2026 the judge in that case wrote that the BOTS Act "unambiguously applies to 'any person' and not just to 'bots.'"
The new ones, side by side
The AI era added several kinds of automation that differ from each other as much as a search crawler differs from a scalper, and I'd lay them out like this.
| What it does | Examples | Who asked, and when | Whose identity | Reads or acts |
|---|---|---|---|---|
| Indexes pages for search | Googlebot, OAI-SearchBot | No particular person; the company runs it | Its own published name | Reads |
| Collects pages for AI training | GPTBot, ClaudeBot | No particular person; the company runs it | Its own published name | Reads |
| Fetches a page someone asked about | ChatGPT-User, Claude-User | A person, just now | Its own published name | Mostly reads |
| Works inside your browser | Claude in Chrome | A person, now or on a schedule they set | Your browser session and logins | Reads and acts |
| Works from the company's cloud | Muse, dots, Grok Bot, Claude Duet, Google-Agent | A person, not always in the moment | Varies by product; Meta says Muse appears as you | Reads and acts |
| Buys or breaks things | Scalper and credential-stuffing scripts | Whoever runs it | Many, fake or stolen accounts | Acts |
The first two rows both collect pages, but they offer a site very different deals: search is supposed to send readers back, and training mostly doesn't. Cloudflare publishes crawl-to-referral ratios that show how lopsided the exchange can be: counting all of Anthropic's bots, it crawled about 55,000 pages on Cloudflare's network for every visit it referred back in December 2025, and about 483 over the 28 days to Oct. 23, 2026 (Cloudflare Radar). Those are totals over a period, not counts per question.
The next three rows are where this year's fights are. Since July, Cloudflare has put chat fetchers and browser-driving agents into a single "Agent" category, which it defines as automation "acting, usually in real time, on a person's behalf." A fetcher shows up because someone just asked about a page; an agent can also click, sign in and buy. The AI companies treat fetchers differently from crawlers when it comes to robots.txt. OpenAI says that for ChatGPT-User "robots.txt rules may not apply" because a user started the action. Google says its user-triggered fetchers "generally ignore robots.txt rules," and Perplexity says the same of Perplexity-User, while Anthropic says "Anthropic's Bots respect 'do not crawl' signals." Perplexity fought with Cloudflare over exactly this last year. Cloudflare accused it of stealth crawling with undeclared user agents and delisted it as a verified bot. Perplexity said Cloudflare had misattributed Browserbase traffic and argued, "This is the fundamental difference between a user agent and a bot."
Agents that act raise the identity question. Claude in Chrome "shares your browser's login state," so the site sees your signed-in session. Meta says that when Muse browses the Internet, "it will appear as your activity," even though Muse runs on its own cloud computer. OpenAI's documentation for dots suggests that if a site blocks the cloud browser, you can ask your dot to try your connected computer instead. Grok Bot, which I use most days, shows up in Cloudflare's directory of verified bots with the user agent GrokAgent and a published signing key, though I couldn't find where SpaceXSI, the company behind it, documents that. My data-broker cleanup with it last month still kept stopping at CAPTCHAs that needed me. A residential proxy, which routes traffic through ordinary home Internet connections, or a cloud browser can carry any of these rows, so a visitor's IP address doesn't tell you its purpose.
Some sites are pushing back. Amazon blocked Muse in September and said agents "should operate openly." In August the Ninth Circuit said of Perplexity's Comet browser that "it is the user who 'accesses' Amazon's computers," but only at the preliminary-injunction stage and only on the computer-fraud claims Amazon brought, and the court noted that Amazon can still restrict access through its terms of service. It added, "We do not establish a new legal regime governing agentic AI."
One question, many requests
One question can also turn into a lot of requests. Google describes AI Mode's "query fan-out technique" as "issuing a multitude of queries simultaneously on your behalf," although those queries run against Google's own index and don't all reach a website. OpenAI and Perplexity say their deep research tools read "hundreds" of sources for a question, and Gemini's reads "up to hundreds of websites." An October preprint based on production ChatGPT traces found GPT-5.4 Thinking issuing about 12 queries per run, surfacing about 74 candidate URLs and citing about 8. Not all of my 480 tool calls fetched a web page, but the shape is the same: one question from me and a crowd of automated visits on my behalf. Cloudflare's version is that agent traffic "grows with how many questions people ask, not how much you publish," and it says daily requests from AI agents on its network grew more than 1,700% in a year.
The headline numbers need the same care. When Prince wrote in June that bots had "passed human traffic online for the first time in the Internet's history," he meant Cloudflare's count of requests for HTML pages, the pages themselves. For Oct. 16 to 23, bots made 66.6% of HTML requests on Cloudflare and 39.5% of all requests (Radar). One likely reason for the gap is that a person's browser also requests a page's images, scripts and other files, while many bots only want the page.
Under Prince's "synonyms" post, Ed Zitron asked whether that meant "overstating the presence of ai 'agents' vs regular bots," which is the right question to ask of any of these numbers. For the same October week, Radar counted 7.7% of HTML requests from bots that declare an AI purpose and 54.8% from bots it classes as non-AI, though an agent that looks like an ordinary browser won't show up in the AI count. Each number is one company's view of its own network, and the companies measuring bots also sell the tools for managing them. Prince has projected that automated traffic could reach about 1,000 times human traffic within five years, with what he called "the giant caveat that I've gone wrong so far in every prediction that I've made on this."
Better questions than "is it a bot?"
Cloudflare made the point well in April: "There are wanted bots and there are unwanted humans." These are the questions I'd ask about any automated visitor:
- Who operates it, and can you verify that? A user-agent header is just the name a piece of software gives itself, and anyone can type one. A verified signature can show which company sent it.
- Did a person ask for this just now, earlier on standing instructions, or not at all?
- Whose identity is it using: its own, your session in your own browser, or a cloud browser signed in as you? A security team would handle each of those differently.
- Is it only reading, or is it changing something, like a cart, an account, a post or a payment?
- Does it slow down or stop when the site asks, and who answers for it if it doesn't?
Underneath all five, a site is also asking whether anything comes back, like a visitor, a sale or a payment. The hardest case is the third question answered with "yours," because sharing my login doesn't tell the site what the agent is allowed to do. That's the bank problem I wrote about last week in A Password Is Not Permission.
What's changing, and what I'd do
Sites and agent builders are working on this from both ends. OpenAI says ChatGPT Work's cloud browser signs its requests with Web Bot Auth so sites can verify the traffic came from OpenAI, Google signs "a subset" of Google-Agent's requests and calls that "experimental," and Claude Duet hasn't published a method. Web Bot Auth is an IETF Internet-Draft, signing is optional, and the working group's charter leaves authenticating the end user out of scope by design. Even when a trusted signature tells a site which company sent an agent, the site still has to decide what that agent may do, including when it also knows who is logged in.
Cloudflare also built its newer detection because bots now "use real browser environments, and pass individual CAPTCHAs," and this month it said that identity alone isn't enough: "trust depends on what it does next." It's scoring, internally for now, whether verified bots honor robots.txt and slow down when a site sends an HTTP 429 "Too Many Requests" response. The other option is charging. Cloudflare is testing ways for sites to charge bots for access in closed betas, and it announced Wallets for agents in August.
If I were running security or a website for a company, I'd stop treating automated traffic as one "bot" number and look at it by what it does. I'd separate claimed identity from verified identity, decide which agents I actually want (a customer's shopping agent might be one), and check that my rules treat verified bots the way I intend. For the agents I use myself, I want to know which row of that table I've turned on.
Which of these rows shows up most in your traffic or your own work, and which would you let in? Tell me on X, LinkedIn, or at contact@exagentica.ai.