AI crawler tracking methods compared
| Method | Cost | Setup | Verifies the crawler | Best for |
|---|---|---|---|---|
| 1. Server access logs | Free | Read the log files you already have | Only if you check IPs yourself | Anyone with log access |
| 2. Verify against published IP lists | Free | A script or a lookup per request | Yes, that is the method | Telling real GPTBot from fakes |
| 3. Cloudflare AI Crawl Control | Every Cloudflare plan | Site behind Cloudflare | Cloudflare's own bot detection | Sites already on Cloudflare |
| 4. Your host's or CDN's logs | Depends on the host; Cloudflare Logpush needs Enterprise, Vercel Log Drains need Pro | Export or stream the request log | Only if you check IPs | Sites on Vercel, Netlify and similar |
| 5. Known Agents (formerly Dark Visitors) | Free up to 100K events a month; Startup $29 | Send visit events to its API | Not stated on its pricing page | A hosted agent analytics dashboard |
| 6. Promvia (ours) | Edge worker on every plan, Free included; CDN log connectors on paid plans | A Cloudflare Worker, a Vercel Log Drain or Cloudflare Logpush | Yes, for CDN log and edge worker hits in range | Crawler visits beside AI citations and AI-referred visits |
Search Console does not belong in this table: its crawl stats cover Google's own crawlers, and Google-Extended, the token that controls Gemini's use of your content, never appears as a visitor at all. It is a robots.txt token read by Googlebot.
Which AI crawlers to look for
Each AI company runs more than one agent, and they do different jobs. The ones that matter most for being cited are the search crawlers and the answer-time agents, not the training crawlers.
- Training crawlers collect pages for model training: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Bytespider (ByteDance), Meta-ExternalAgent (Meta).
- Search crawlers index pages for an assistant's own search: OAI-SearchBot (ChatGPT search), Claude-SearchBot (Anthropic), PerplexityBot (Perplexity).
- Answer-time agents open a page while answering a user's question: ChatGPT-User, Claude-User, Perplexity-User. A visit from one of these is the closest a log gets to "an assistant read this page to answer someone".
- Google's AI Overviews and AI Mode use Googlebot, so they are invisible as a separate agent; Apple uses Applebot.
The full table, with each operator's robots.txt token, purpose and published IP list, is in AI crawler user agents, verified.
1. Read your server access logs
Every request to your site is already in your web server's access log, with the user agent in it. Searching that log for the agent names above shows which AI crawlers came, which pages they asked for, and when. On a server you run yourself, a single search for the names across the log files is enough for a first answer.
- Count per agent and per page, not just in total. Which pages ChatGPT-User opens tells you which pages ChatGPT reads when it answers.
- Look at the status codes. A crawler that receives 403 or a challenge page is being turned away, usually by a firewall or bot protection, whatever robots.txt says.
- Keep at least a few weeks. Crawler visits come in bursts; one day of log says little.
2. Verify each crawler against its published IP list
A user agent is a claim. Uptime monitors, SEO tools and scrapers send "GPTBot" when it suits them, so a count of user agents is a count of claims. OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple and Common Crawl publish the IP ranges their crawlers use; a request is real when its source address falls inside the range for the agent it names. Meta and ByteDance publish no list for their AI crawlers, so their visits cannot be verified this way.
On our own site, from 26 August to 5 October 2026, 16 of 68 requests claiming to be GPTBot failed the IP check, and all 16 were our own weekly crawler-access check; the other 52 all passed. PerplexityBot looked the same: 9 of 34 failed, all 9 ours. The fakes in your log may be your own tools, so mark your own monitoring before you count. The per-agent figures are in what our own log showed.
3. Use Cloudflare's AI Crawl Control
If your site is behind Cloudflare, its dashboard already watches AI crawlers. AI Crawl Control, formerly called AI Audit, shows which AI services access your content and their request patterns, flags crawlers that ignore your robots.txt, and lets you allow or block each crawler. It needs no logs and no code.
What it does not do is join the visits to anything else: it shows that ChatGPT-User came, not whether ChatGPT then cited the page, or whether the answer sent you a visitor. Cloudflare also blocks AI crawlers by default on new domains, so check its settings before reading a quiet chart as a lack of interest.
4. Use your host's or CDN's request logs
On a managed host there is no log file to open, but the request log exists and can be exported or streamed. On Cloudflare, Logpush sends HTTP request logs, but only on the Enterprise plan; a Worker can forward requests on any plan. On Vercel, a Log Drain streams every request your site serves to an endpoint you choose, on a Pro or Enterprise team, and Vercel bills the volume. Once the lines arrive somewhere you can query, the method is the same as steps 1 and 2.
5. Use a crawler analytics tool: Known Agents
Known Agents, which Dark Visitors' address now redirects to, is a hosted dashboard for AI agent and crawler visits. Your site sends it visit events through its API, and it counts and names the agents.
6. Promvia (our product): crawler visits beside citations and AI visits
Read this section as the maker's account. Promvia reads AI crawler visits from a Cloudflare Worker (on every plan, Free included) or from a Vercel Log Drain or Cloudflare Logpush connector (on paid plans), and marks a visit Verified only when it came through one of those authenticated routes and its address is inside the operator's published range. Anything else stays Detected. The same dashboard shows which AI answers cited your pages and the visits AI assistants sent, so you can see whether a page ChatGPT-User opened was then cited.
- Where it is behind: Bingbot, Bytespider and Meta's agents stay Detected, never Verified, because we do not load Bing's list yet and ByteDance and Meta publish none.
- It does not block crawlers; that stays with your CDN or robots.txt.
- Plans and prices are on our pricing page.
What crawler visits can and cannot tell you
A crawler visit is a precondition, not a result. An engine cannot cite a page it never fetched, but most fetched pages are never cited. When we matched every answer that cited our site to our crawler log, answer-time fetches just before the answer were rare; the search crawler's earlier visit to the same page was nearly always there. Joining server logs to AI citations has the numbers and the rows.
- No visits from an assistant's search crawler: check robots.txt and your firewall first. Auditing robots.txt and your CDN for AI crawlers goes through both.
- Visits but no citations: the page is in the pool but not chosen. That is a content and authority problem, not a crawling one.
- Citations but no visitors: answers name you without sending clicks, or the clicks arrive without a referrer. Tracking AI traffic to your website covers the visits half.
Frequently asked questions
How do I know if ChatGPT is crawling my website?
Look for OAI-SearchBot, ChatGPT-User and GPTBot in your access logs, and check that each request comes from an address in OpenAI's published IP ranges. OAI-SearchBot indexes pages for ChatGPT search, ChatGPT-User opens a page while answering a user, and GPTBot collects pages for training.
What is the best tool to track AI crawlers?
If your site is on Cloudflare, start with its AI Crawl Control, which is on every Cloudflare plan. If you want the visits checked against the operators' IP lists and joined to AI citations and AI-referred visits, use a tool that does both; Promvia (ours) does, from a free plan. If you have log access and time, your own logs plus the published IP lists cost nothing.
Can I see AI crawlers in Google Analytics?
No. Crawlers do not run the JavaScript that Google Analytics depends on, so their visits never reach it. Google Analytics shows the people AI assistants send, not the crawlers; crawler visits are in your server or CDN logs.
Should I block AI crawlers?
Block training crawlers if you do not want your content used for training; that does not stop an assistant from citing you. Blocking the search crawlers and answer-time agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot) removes your pages from those assistants' answers.