The Internet Wasn't Built for a Billion Agents

To begin researching this article, I started with eight AI agents that scraped more than 200 web pages in just a few minutes. Those sites absorbed thousands of backend operations they will never be paid for. No ad impressions. No subscriptions. No traffic sent back.

I am the problem I'm about to describe.

The click that became a thousand

When you ask ChatGPT or Perplexity a question, it feels like one request. It isn't. Google's AI Mode breaks a single prompt into multiple sub-queries. OpenAI's Deep Research browses roughly 100 web pages per question. Perplexity reads 200+ sources per report.

Stack the layers and it gets ugly.

A traditional Google search creates one unit of backend load. AI-powered search can be 10x. Reasoning-heavy models: 30-100x. Deep research: 100-500x. A coding agent like Claude Code: 500-1,000x. Multi-agent systems where agents spawn other agents: 1,000-5,000x.

The exact multiple varies by product, but the direction is obvious: one user action now fans out into many more requests, fetches, and compute cycles than the web was designed for. Those ratios come from platform documentation, IEA energy measurements, and ICLR 2025 token consumption research. The energy numbers point the same way. One ChatGPT query uses 2.9 watt-hours. A Google search uses 0.3.

The infrastructure on the other end was built for humans. Human traffic has a rhythm: it ramps in the morning, dips at lunch, falls overnight, and follows the sun around the planet. Capacity planning assumed some version of that pattern.

AI agents do none of that. They do not sleep, pause, or smooth demand naturally. They run at machine speed. Spencer Kimball, Cockroach Labs' CEO, says a single agent can generate 5,000 actions per second. In a 2026 survey of 1,125 senior cloud architects, 83% said they expect AI to cause infrastructure failure within two years.

Taking without giving back

Ten years ago, the bargain was simple. You let Google index your pages. Google sent traffic back. That traffic paid some bills.

That deal is dead.

Google now crawls 18 pages per visitor it sends back. Microsoft sits at 40:1. OpenAI: 1,500:1. Anthropic, the company behind Claude (yes, the AI I used to write this): 40,000:1. At peak, ClaudeBot hit 500,000 pages crawled per referral. Half a million pages scraped to send back one visitor. Let that sink in.

Cloudflare Radar crawl-to-refer ratios showing Anthropic at 40,000:1 vs Google at 18:1 — Source: Cloudflare Radar

Cloudflare CEO Matthew Prince (my former boss) called Google "the great patron of the Internet for the last 27 years" and "the great villain of the Internet today." In the second half of 2025, Cloudflare blocked 416 billion AI bot requests. 85% of customers who take action on AI crawlers choose to block them entirely.

Bots now account for 51% of all web traffic, according to Imperva's 2025 Bad Bot Report. Humans are the minority on their own Internet.

Even when publishers try to help, the economics still look ugly. In a March 5, 2026 post, Drupal founder Dries Buytaert said that after making every page on his site available as Markdown and watching the results for a month, AI crawlers fetched 1,241 pages for every citation they sent back. He also reported that no crawler used content negotiation, bots fetched both HTML and Markdown versions, crawler traffic rose 7%, and llms.txt saw no adoption. The older mechanism here is the Robots Exclusion Protocol, which sounds like a joke but is the real name: a voluntary standard from the 1990s that the IETF only codified in RFC 9309 in September 2022.

Small operators absorb the worst of it. Triplegangers, a seven-person company selling 3D scans, went dark for days after OpenAI's GPTBot hit them from 600 IP addresses, scraping all 65,000 products. Their AWS bill spiked and they had no recourse until they set up blocking. SourceHut, a git hosting platform, blocked entire cloud providers. A 404 Media survey found 27 of 43 cultural institutions attribute outages to AI bots. Open-source projects like GNOME and KDE report 97% bot traffic.

I should be fair here. Many of the loudest traffic numbers come from companies that sell bot protection. They benefit when the problem feels urgent. But the independent reporting points the same way. The 404 Media survey is independent journalism. The open-source project reports are first-hand. The Triplegangers case is documented by TechCrunch. The direction is real even if some of the biggest numbers come with vendor incentives attached.

The canary is dead

High-volume events are where the future arrives early. They show what normal traffic looks like once agents scale.

Germany's DFB Cup Final ticket sale: 160 million bot requests versus 1.66 million from humans. 96 bots for every human. Taylor Swift's Eras Tour threw 3.5 billion requests at Ticketmaster. Four times their previous record. Congress got involved.

Akamai called Black Friday 2025 "the first agentic cyber week." AI agent commerce traffic grew 6,900% year-over-year. They counted 11.8 billion bot requests on Black Friday alone.

Here's the part that should worry infrastructure teams: those peak events preview the baseline. Once AI agents are handling shopping, booking, price comparison, and research for hundreds of millions of users, every day looks like Black Friday to the backend.

Databases are already cracking

This part is not hypothetical. These are production failures.

September 2025: Clerk's PostgreSQL connection pool hit a thundering herd. A routine version upgrade synchronized connection recycling across the fleet. Four days of degraded service. The fix was adding randomized jitter. Every database engineer knows that technique. Almost nobody implements it until something breaks.

July 2025: Replit's AI agent dropped a production table during a code freeze. More than 1,200 executive records were gone. The agent then fabricated test results and claimed rollback was impossible.

Late 2025: Amazon's AI coding tool Kiro caused a 13-hour AWS outage by autonomously deleting and recreating a production environment. Employees said it wasn't the first time.

These are different failures, but they rhyme: more autonomous actions, more concurrency, more opportunities for one mistake or one synchronized burst to cascade.

"Just auto-scale" is the easy answer. I don't buy it.

PostgreSQL forks a new process for every connection. Connection storms do not get easier just because demand is machine-generated. OpenAI runs ChatGPT for 800 million users on a single-primary PostgreSQL instance with 50 read replicas, and it still had to move write-heavy workloads out of Postgres entirely. If OpenAI needs architectural workarounds, your startup probably will too.

Some bottlenecks sit outside the database entirely. Payment processors have fixed rate limits. Inventory systems have fixed rate limits. Your database might scale. The systems it depends on will not.

Then there is the thundering herd. A cache key expires. A thousand agents notice the miss at the same moment. They pile into the database in one burst. Baseline to catastrophe in a second. Auto-scalers have not even fired yet.

Jevons Paradox is eating the Internet

This is the part people keep missing. The strain is not proof that AI is failing. It is proof that AI is useful.

According to Stanford's 2025 AI Index, the inference cost for a system performing at the level of GPT-3.5 dropped more than 280-fold between November 2022 and October 2024. In the same report, the share of organizations saying they use generative AI in at least one business function jumped from 33% in 2023 to 71% in 2024. Jevons spotted this with coal in 1865: make something cheaper and more efficient, and people usually use much more of it.

The productivity gains driving that demand are not imaginary. A BCG-Harvard trial with 758 consultants: 25% more output, 25% faster, 40% higher quality. GitHub Copilot: 55% faster task completion. Klarna's AI handles two-thirds of customer service chats, does the work of 853 people, saves $60 million. Adobe found AI-referred shoppers convert at 4.4x the rate.

Every one of those wins drives more adoption. More adoption drives more agent traffic. More agent traffic drives more database queries, more API calls, and more bandwidth consumed by entities that never see an ad or buy a subscription.

This cycle has happened before. The iPhone launched and AT&T's data traffic jumped 5,000%. The network crumbled. But the traffic represented real value, and the infrastructure caught up. The $500 billion telecom overbuild of the late 1990s, widely mocked as wasteful, became the backbone of cloud computing.

AI infrastructure will catch up too. It always does. The real question is who absorbs the damage in the gap between now and then.

The solutions exist. The clock is the problem.

Some of the fixes are already on the table. Pay Per Crawl uses HTTP 402, a status code that sat mostly untouched for 30 years, to let publishers charge for crawled pages. Stack Overflow became a launch partner in February 2026. Web Bot Auth gives bots cryptographic identity so publishers know who is scraping. AWS, Akamai, Visa, and Shopify are implementing it.

Caching and local inference can help too. The more often an agent can reuse prior work or answer a request on-device, the less traffic reaches origin infrastructure in the first place.

Firecrawl sits in an awkward spot. It makes scraping easier, which is part of the problem. It also makes scraping more efficient, which is part of the solution. One clean extraction is better than an agent hammering the same page ten times. It has built-in rate limits and robots.txt compliance. Same argument as CDNs. They did not create the bandwidth problem, but they did help channel chaotic traffic into something manageable.

More aggressive tools are showing up too. Scrapling, an open-source Python scraping framework, advertises adaptive selectors, MCP integration, and fetchers that can bypass Cloudflare Turnstile. Last week, WIRED reported that OpenClaw users were pairing Scrapling with agents to scrape protected sites, and that Cloudflare was already working on new mitigations. That is the cat-and-mouse dynamic now: better scraping tools, better defenses, more load in the middle.

The deeper problem is economic. The Internet ran on a simple swap: you let me index your content, I send you visitors. AI broke the swap. It indexes the content and keeps the visitors.

The gap is where the damage happens

I keep coming back to the same irony. This article exists because AI agents scraped 200+ web pages on my behalf. Those sites paid the compute bill. I got the value. That arrangement is obviously temporary.

The Big Five are spending over $600 billion on AI infrastructure in 2026. Goldman Sachs projects data center power demand surging 165-175% by 2030. NIST opened a formal request for information on AI agent security on January 12, 2026. Cornell researchers estimate AI will produce 24-44 million metric tons of CO2 annually by 2030.

The infrastructure will catch up. It always does.

But the seven-person company that got DDoS'd by a training crawler in January 2025 cannot wait for 2030. The open-source maintainer serving 97% bot traffic does not have venture funding. The cultural institution that went dark because AI bots swamped its server does not have a Cloudflare enterprise plan.

The race is real. Right now, the agents are winning, and smaller operators are paying the bill.

Update (March 10, 2026): Cloudflare introduced a /crawl endpoint that may help address at least some of these challenges.

Sources

John Engates writes about agentic AI and the infrastructure it keeps breaking at exagentica.ai.