AI search

AI Crawler Traffic: What It Does to Your Analytics Numbers

AI crawler traffic now outweighs human visits on much of the web. How it skews pageviews, how it differs from AI referrals, and what to block or allow.

By Jasper Koers Published 6 min read

On this page
  1. What Cloudflare reported about AI crawler traffic
  2. AI crawler visits vs AI referral visits
  3. What AI bots do to your analytics
  4. Bot filtering in analytics: keeping the numbers clean
  5. Should you block AI crawlers?
  6. What this means for your analytics

On 1 July 2026, exactly one year after it started blocking AI training crawlers by default for new domains, Cloudflare published Content Independence Day, one year on. The headline numbers are stark: 52% of crawler requests on Cloudflare's network were for AI training in June 2026, up from 22% in spring 2025, and more than 50% of all internet traffic is now non-human. Crawlers that mix search, agent and training purposes accounted for more than 36% of crawler activity. In some of the most heavily crawled categories, Cloudflare says human traffic fell by as much as 40% in less than a year.

On the same day Cloudflare shipped new controls that let every customer, Free plan included, block AI bots by what they do: Search, Agent or Training. For anyone who reads a traffic dashboard, this raises two questions. How much of what you are counting is actually human? And which AI crawler traffic should you block without disappearing from AI search?

What Cloudflare reported about AI crawler traffic

Cloudflare sits in front of a large share of the web, so its numbers are a reasonable proxy for what an average site sees. The key figures from the July 2026 report:

Metric Figure
Crawler requests for AI training, June 2026 52% (spring 2025: 22%)
Crawler activity from mixed-purpose crawlers (search + agent + training) more than 36%
Share of internet traffic that is non-human more than 50%
Drop in human traffic for some heavily crawled categories up to 40% in under a year
Google's share of referral traffic about 88%

Two things stand out. First, training crawlers take content without sending anyone back, and they now make up the majority of crawler requests. Second, more than a third of crawler activity comes from bots that do several jobs at once, which makes it hard to allow one purpose (search) while refusing another (training). Cloudflare says it wants to bring that mixed-use traffic down to zero within a year. The report names retail, software, IT and finance among the affected categories but does not give per-category numbers.

AI crawler visits vs AI referral visits

These two get confused all the time, and they mean opposite things for your business.

  • An AI crawler visit is a bot fetching your page: GPTBot collecting training data, Claude-SearchBot indexing for search, or an agent such as ChatGPT-User fetching a page because a user asked a question. No human sees your site.
  • An AI referral visit is a person who read an answer in ChatGPT, Perplexity, Claude or Gemini and clicked through to your site. That is a real visitor, and often a well-qualified one.

The first inflates or pollutes your numbers. The second is the traffic you want to grow and measure. We wrote a separate guide on how to track AI and ChatGPT traffic to your website; this article is about keeping the bots out of that picture.

The major AI companies publish separate user agents per purpose, which is what makes selective blocking possible:

Purpose OpenAI Anthropic Google
Training GPTBot ClaudeBot Google-Extended (robots.txt token only)
Search indexing OAI-SearchBot Claude-SearchBot Googlebot
Fetching for a user (agent) ChatGPT-User Claude-User —

Sources: OpenAI, Anthropic, Google. Google states that Google-Extended "does not impact a site's inclusion in Google Search."

What AI bots do to your analytics

How much AI crawler traffic lands in your reports depends on how your analytics collects data.

Server logs and log-based analytics see everything. Every crawler request is a line in the log, so without strict filtering, a rising crawler share means rising "visits" that no human made.

Script-based analytics (a JavaScript snippet in the page) only counts visitors whose browser runs that script. Many crawlers fetch the raw HTML and never execute JavaScript, so they never show up. But that protection is getting weaker: agents that drive a real or headless browser do run scripts, and some identify themselves while others do not.

The symptoms of AI bots slipping into your data are recognisable:

  • Spikes of direct traffic with no referrer, landing on one page and leaving
  • Sessions from data-centre locations rather than where your audience lives
  • Pages that are rarely linked to suddenly getting steady traffic
  • Bounce rate and time on page moving without any change to the site
  • Traffic from languages or regions you do not serve

If those patterns look familiar, read why your analytics data is probably wrong for the wider list of causes.

Bot filtering in analytics: keeping the numbers clean

Good bot filtering works in layers, because no single check catches everything:

  1. User-agent lists. Drop requests from known bot and crawler user agents, and keep the list current. AI companies add new agents regularly.
  2. Headless and automation detection. Catch HeadlessChrome, Playwright, Puppeteer, Selenium and similar tools that announce themselves.
  3. Data-centre IP ranges. Real visitors rarely browse from cloud provider networks; automated traffic often does.
  4. Behavioural signals. Flag visitors who load hundreds of pages a day, or patterns that look like referral spam.
  5. Filter at read time where possible. If you mark traffic as suspicious instead of deleting it, you can always check what the filter is hiding.

What you should not filter out is the AI referral visit. A human who clicks from ChatGPT should be counted, attributed to ChatGPT and kept apart from both the bots and your "direct" bucket.

Should you block AI crawlers?

Blocking is a business decision, not a technical one. A practical way to think about it:

Crawler type What you get What you give Typical choice
Training (GPTBot, ClaudeBot, Google-Extended) Nothing direct Your content in future models Block if your content is your product
Search (OAI-SearchBot, Claude-SearchBot, Googlebot) Citations and clicks in AI answers Indexing Usually allow
Agent (ChatGPT-User, Claude-User) A user getting an answer based on your page A fetch per question Allow unless it hurts you (e.g. ad-funded pages)

Blocking AI search crawlers means AI assistants are less likely to cite or link you. OpenAI says sites that opt out of OAI-SearchBot won't appear in ChatGPT search answers, and Anthropic says blocking Claude-SearchBot may reduce visibility in search results. If AI visibility matters to you, see our piece on generative engine optimization.

robots.txt vs Cloudflare's AI crawler controls

robots.txt is the polite route: it works for crawlers that respect it. Anthropic says its bots honour robots.txt. OpenAI notes that for ChatGPT-User, which acts on a user's request, robots.txt rules "may not apply". A simple setup that refuses training but keeps search might look like this:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Cloudflare's controls enforce at the network edge instead of asking nicely. Since 1 July 2026 you can set Search, Agent and Training separately to block on all pages, block only on pages that show ads, or allow. From 15 September 2026, new domains onboarding to Cloudflare will block Training and Agent bots on ad-supported pages by default, while Search stays allowed. Cloudflare notes that multi-purpose crawlers combining search and training will be affected by the training block. Existing customers can opt out of the new defaults in the dashboard.

The catch is that mixed-purpose crawlers make clean choices hard: blocking training can also block a crawler you need for search. That is exactly the problem Cloudflare says it wants to solve over the next year.

What this means for your analytics

With more than half of web traffic non-human, a pageview count without serious bot filtering is a guess. The goal is two clean numbers: real humans, and the share of them that arrived from AI assistants.

ClearAnalytics rejects known bot user agents, headless browsers and requests from major cloud-provider IP ranges before they become pageviews, flags suspicious visitors instead of silently counting them, and reports AI referrals from ChatGPT, Perplexity, Claude and Gemini as their own source. It does this without cookies and with data stored in the EU. See how we track AI visibility if that is the number you care about.

Frequently asked questions

How much website traffic comes from AI crawlers?

Cloudflare reported in July 2026 that more than half of all internet traffic is non-human and that 52% of crawler requests on its network were for AI training in June 2026. The share on your own site depends on how often crawlers target your content.

Do AI crawlers show up in Google Analytics or other analytics tools?

Many AI crawlers fetch the HTML without running JavaScript, so they never reach a script-based analytics tool, but crawlers and agents that use a real browser can. Tools differ in how well they filter those, so check your bot filtering rather than assuming it.

Should I block AI crawlers?

Block training crawlers if you do not want your content used to train models, but think twice before blocking AI search crawlers, because they decide whether you can be cited and linked in AI answers.

What is the difference between AI crawler traffic and AI referral traffic?

AI crawler traffic is automated fetching by bots such as GPTBot or ClaudeBot. AI referral traffic is real people clicking through from an AI assistant to your site, which is the traffic you actually want to measure.

What did Cloudflare change for AI crawlers on 1 July 2026?

Cloudflare split its AI bot controls into Search, Agent and Training categories for all customers, with options to block on all pages, only on pages with ads, or not at all. From 15 September 2026 new domains block Training and Agent bots on pages with ads by default.

  • ai-crawlers
  • bot-filtering
  • ai-search
  • cloudflare
  • data-quality

Back to blog Read as markdown

Enjoyed this article?

Try ClearAnalytics for free and get privacy-first analytics for your website.