AI crawler log analysis: what GPTBot and ClaudeBot actually fetch

Spoiler for the scrollers: the server access log is the only place where “the AI bots see my content” stops being a hope and becomes a line with a status code on it.

I open a raw access log the same way every time — filter for GPTBot, then ClaudeBot, then PerplexityBot — and, more often than not, the same thing happens: robots.txt said one thing, the sitemap promised another, and the log tells a third story. Everything upstream of the log is intent; “I added llms.txt” is intent too. The log line is evidence: it names the bot, the URL, the status code and the byte count, and once those lines are filtered, four questions answer themselves. Are the bots reaching the money pages? Are they wasting budget on junk? Are they blocked? And — the one everybody skips — are the requests even real?

The user agents worth filtering for

Not every AI bot does the same job: some crawl in bulk to build a corpus, some fetch a single page the instant a user asks a question, and reading the logs correctly starts with knowing which is which.

User agent	Operator	What it’s doing
GPTBot	OpenAI	Bulk crawl for training/data
OAI-SearchBot	OpenAI	Building the ChatGPT search index
ChatGPT-User	OpenAI	On-demand fetch — a user asked something
ClaudeBot	Anthropic	Bulk crawl for Claude
Claude-User	Anthropic	On-demand fetch inside a Claude chat
PerplexityBot	Perplexity	Indexing for Perplexity answers
Googlebot + Google-Extended	Google	Crawl governed by the Extended token (see below)

A spike in ChatGPT-User or Claude-User on one URL is a live signal — someone is asking an AI about that exact topic right now, and the bot is fetching the page to answer. Honestly, that single line deserves more attention than a mountain of bulk-crawl volume, because bulk crawl says “we might use this someday” while an on-demand fetch says “we’re using it now.”

What one log line tells you

A single access-log line carries four fields that matter for GEO, and I read them in this order:

• Path — which URL was fetched. Aggregate these and coverage falls out on its own.
• Status — 200 (served), 304 (not modified), 403 (blocked), 404 (missing), 5xx (broken). A bot that only ever gets 403s never sees you.
• Bytes — a 200 that returns 800 bytes on a 4,000-word article means the bot got a shell, not the content.
• Frequency + timestamp — how often, and whether recrawl tracks the publish cadence.

The four gaps to hunt

1. Coverage gaps. Hypothesis: the bots see everything in the sitemap. Check: group successful (200) fetches by path and compare against the sitemap. What usually comes out: a tail of pages the bots never request, invisible to AI answers no matter how good they are. Deep, poorly-linked pages are the usual casualties, which makes this a crawl-budget and internal-linking problem, not a content one.

2. Wasted crawl. Count the requests that hit redirects (301/302 chains), 404s, faceted-filter URLs, or infinite pagination — every one of those is budget the bot did not spend on a real page. On a large site this is where most AI-crawl waste hides, and it’s rarely where anyone was looking.

3. Blocked or thin responses. 403s mean something — a WAF rule, a rate limiter, or robots.txt — is turning bots away, and small byte counts on big pages mean the bot fetched HTML that renders empty. Since most AI crawlers do not run JavaScript, a client-rendered SPA reads as a blank page in the logs. Gorgeous in the browser, blank in the log!

4. Verification. Confirm the requests are genuine before trusting any of the above — more on that in a moment, because it’s the step where the whole report can quietly go wrong.

Verify the bot is real

The user-agent string is self-reported and trivial to forge — scrapers routinely wear GPTBot’s name to dodge blocks. So before I draw a single conclusion, I confirm the source IP:

• OpenAI, Anthropic and Google publish the IP ranges their crawlers use, as machine-readable lists to match against.
• Forward-confirmed reverse DNS is the fallback: reverse-resolve the IP, check it lands on the operator’s domain, then forward-resolve that hostname back to the same IP.

Skip this and the “GPTBot coverage” report may be counting a scraper in a GPTBot costume. How likely is that on your site? More likely than it seems, I suspect — the spoofing is cheap and the payoff (dodging a block) is real. Cheap trick, expensive mistake!

The checklist

• Filtered logs for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Googlebot.
• Verified each bot by source IP (published ranges or forward-confirmed rDNS).
• Compared 200-fetched paths against the sitemap — listed the never-crawled pages.
• Counted wasted crawl: redirect chains, 404s, faceted URLs, pagination.
• Flagged 403s and thin responses (small bytes on large pages).
• Checked recrawl frequency tracks your publish cadence.
• Separated on-demand (ChatGPT-User / Claude-User) hits as live-interest signals.

Content pages (200) 45%

Redirects / 404 / facets 40%

Assets & other 15%

source: internal audits, 2026

The split is the point: when nearly half the crawl lands on redirects and dead ends, the fix is not more content — it is cleaning the paths the bots waste budget on. (My subjective take: most “we need more content for AI” conversations should start with this chart, and quite a few would end there.) Our AI crawler log tool does this pass for you: it parses the logs, verifies each bot, and hands back the coverage-gap and wasted-crawl lists.

In short

• Server logs are the only evidence of what AI crawlers actually fetch — the rest is intent.
• Filter for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Googlebot; treat on-demand fetchers as live interest.
• Read path, status, bytes and frequency to find coverage gaps, wasted crawl, and blocked or thin pages.
• Google-Extended is a robots token, not a log line — the fetch is still Googlebot.
• Verify every bot by IP before you trust the numbers; user agents are trivially spoofed.

And if the log says GPTBot has never once fetched your best page — well, now you know, which is already more than the sitemap ever told you.

Sources

• OpenAI, “GPTBot” and “OAI-SearchBot” docs — platform.openai.com/docs/bots: official user agents and published crawler IP ranges.
• Google Search Central, “Overview of Google crawlers and fetchers” — developers.google.com/search/docs/crawling-indexing/overview-google-crawlers: confirms Google-Extended is a robots token applied to Googlebot, not a separate crawler.
• RFC 9309, “Robots Exclusion Protocol” — rfc-editor.org/rfc/rfc9309: the standard governing how user-agent tokens and directives are interpreted.
• Related: why SPAs go dark to AI crawlers and managing AI bots in robots.txt.