How to set up AI crawler log tracking (and what to look for once it runs)

Why crawler logs matter for AI search

An AI visibility tracker tells you whether ChatGPT or Perplexity mentioned you. It can't tell you whether those engines ever read the page you wanted cited. That information doesn't live in the answer. It lives in your server or CDN logs, in the requests AI crawlers make when they fetch your pages.

Without it, a lost citation is a guessing game. Maybe a competitor published something better. Maybe your page started returning errors to bots after a deploy. Maybe a firewall rule quietly blocked a crawler last month. Crawler logs separate the editorial problems from the technical ones, which is why they're the first thing we check when a page drops out of AI answers.

Here's how to set it up, step by step.

Step 1: know which crawlers you're looking for

AI companies run several kinds of bots. Some collect training data. Some fetch pages to build a search index for the assistant. Some fetch a page live because a user's question triggered it. The names change over time, so treat this as a starting list and check each operator's documentation for the current set.

Operator	User agents to look for	Roughly what they do
OpenAI	GPTBot, OAI-SearchBot, ChatGPT-User	Training, search indexing, user-triggered fetches
Anthropic	ClaudeBot, Claude-SearchBot, Claude-User	Training, search indexing, user-triggered fetches
Perplexity	PerplexityBot, Perplexity-User	Search indexing and user-triggered fetches
Google	GoogleOther, plus Googlebot for AI Overviews	Research and product crawls; Overviews draw on Google Search's index
Meta	meta-externalagent	Crawling for Meta AI

One distinction to keep in mind: blocking a training bot doesn't necessarily block the search bot from the same company, and the reverse. If you want to be cited in ChatGPT's search answers, the search-indexing bot is the one that matters most.

Step 2: check that you aren't blocking them by accident

Before tracking anything, confirm the door is open.

Open your robots.txt and look for rules naming the user agents above, or broad disallow rules that catch them. Then check your CDN and firewall. Some CDNs now offer a setting that blocks AI bots in one click, and bot-protection rules can challenge or block crawlers that don't run JavaScript. Ask whoever manages your CDN whether any of these are switched on.

Also check rate limiting. A crawler that gets a 429 response on half its requests is, in practice, partly blocked.

Step 3: find the log source that sees all the traffic

This step is where most DIY setups go wrong. If your site sits behind a CDN, many requests are answered from the CDN cache and never reach your origin server. Origin logs then undercount bot traffic, sometimes badly.

So collect logs from the outermost layer that serves requests. For most sites that's the CDN or hosting platform: Cloudflare, AWS CloudFront, Fastly, Akamai, Google Cloud CDN, Vercel, or Netlify. If you serve directly from your own servers, the web server access log is fine.

Step 4: take a first look by hand

If you have shell access to raw access logs, a single command gives you a rough picture before you set up anything permanent:

grep -Ei "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|PerplexityBot|meta-externalagent" access.log \
  | awk '{print $7, $9}' | sort | uniq -c | sort -rn | head -50

That lists the most requested paths and status codes for AI crawlers (field positions depend on your log format, so adjust $7 and $9 if needed). It's crude, but it often surfaces surprises in minutes: a crawler hammering an old sitemap, a block of 404s on moved pages, or no AI crawler traffic at all on the section you care about.

The manual approach doesn't last. Logs rotate, CDN logs need exporting, and nobody wants to rerun this every week.

Step 5: verify the bots are real

Anyone can put "GPTBot" in a user-agent string. Scrapers do it to slip past bot rules. Before you draw conclusions, verify a sample of requests. Several operators publish the IP ranges their crawlers use; a request claiming to be a known AI bot from outside those ranges is probably an impostor.

A good tracking tool handles this for you. If you're building your own pipeline, it's the step people skip and later regret.

Step 6: read your first real report

Once logs are flowing, look at four things in this order.

Coverage of the pages that matter. List the pages you most want cited: key product pages, comparison pages, guides. Check whether each AI crawler has fetched them in the last 30 days. Pages that never get fetched can't be cited from a fresh crawl.

Errors. Filter for 4xx and 5xx responses to AI crawlers. A redirect chain or a soft 404 that humans never notice can make a page invisible to bots.

What the crawler actually received. Many AI crawlers fetch raw HTML and don't run JavaScript. If your content renders client-side, the crawler may be getting an empty shell. Compare the response size for bot requests with the size of the fully rendered page.

Frequency and recency. A page that was fetched weekly and suddenly stops getting visits is an early warning, often weeks before a citation drops.

Step 7: connect crawls to citations

The final step is the one that turns logs into decisions. For each prompt where you lost or never earned a citation, ask: did the relevant crawler fetch the page that should have been cited? If not, fix access, rendering, or internal links. If it did, and fetched it cleanly, the problem is the content or the sources the engine prefers, and your citation data tells you which.

Doing this by hand means joining two datasets that live in different tools. That's the main argument for a platform that holds both.

Tools that read AI crawler activity

Tool	What's published on our listing	Where it starts
Promptwatch	Agent Analytics: real-time AI crawler logs, crawl-to-citation path, error tracking, alongside prompt and citation data	Professional, $245/mo (25M crawler logs); every agency plan
Scrunch AI	Crawler traffic analytics as part of its page-serving layer for AI crawlers	Plan placement not published; Starter from $250/mo annual
Qwairy	Crawler analytics	Business, €449/mo
Encited	Crawl log analysis with its edge pre-rendering proxy	Essential, $39/mo annual ($49 monthly)

How we'd set it up in Promptwatch

For most teams, we'd do this in Promptwatch, because it's the one place on that list where crawler logs sit next to the prompts and citations you're trying to explain.

Agent Analytics connects to the log source from step 3. Supported integrations are Cloudflare, AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, and a custom HTTP option for anything else. Pick the one that matches where your traffic is served. Once connected, it tracks ChatGPTBot, ClaudeBot, PerplexityBot, GoogleOther, and the Meta AI crawler in real time, flags errors, and shows the crawl-to-citation path, which is step 7 done for you.

Volume is set by plan. Professional at $245 a month includes 25M crawler logs. Business at $579 includes 100M. Agency plans include 10M (Kick-off, $199), 25M (Growth, $399), or 100M (Scale, $799). Essential doesn't include crawler logs, so a team that needs them should start on Professional, which also has a 7-day trial.

Then use the rest of the platform to close the loop. Citation analytics show which sources replaced you when the crawl looks healthy. Visitor analytics show whether fixing a crawl problem brought AI-referred visitors back.

FAQ

Should I block AI crawlers?

That's a business decision. Blocking training bots is common; blocking search and user-triggered bots usually means you won't be cited in those assistants' answers. Decide per bot, not with one blanket rule.

Do I need CDN logs, or are server logs enough?

If a CDN serves cached pages, server logs miss those requests. Use CDN logs where you have them.

Which Promptwatch plan includes crawler logs?

Agent Analytics starts on Professional ($245/mo, 25M crawler logs) and is on every agency plan. Business includes 100M crawler logs.