Methodology
How we measure
AI answers change: the same question can get a different answer a few minutes later. This page explains exactly how Promvia checks, so you know what each number means before you act on it.
Maintained by the Promvia team · Last updated
- Perplexity
- ChatGPT
- Claude
- Google AI Overviews
- Google AI Mode
Optional on paid plans: Gemini and Grok. Not measured yet: Microsoft Copilot. Coming soon: Meta AI. The Bing stand-in that once filled a Copilot column was suspended on 2026-09-22 (see What we actually ask).
Product names and logos belong to their owners.
01
What we actually ask
In plain words: we ask each AI assistant through the connection its maker offers to developers, with web search switched on, the same way for every customer. The app on your phone can answer a little differently.
We call each AI assistant's own API with web search on. ChatGPT: OpenAI's Responses API with the web_search tool forced on every call (model gpt-5.6-luna). Claude: Anthropic's Messages API with its web search tool, up to three searches per answer (model claude-haiku-4-5-20251001). Gemini: the Gemini API with Google Search grounding (model gemini-3.1-flash-lite). Perplexity: its Agent API (preset "fast"), where we count the search results the answer cites rather than every page it searched. The model or preset is frozen into each run when it is queued, so one run never mixes two models. Grok, when an account adds it, is asked through xAI's API with web search on, capped to a single search round (model grok-4.20-0309-non-reasoning). Google AI Overviews and AI Mode are read from Google's results page, including the follow-up request Google sometimes requires. Microsoft Copilot is not measured yet. Until 2026-09-22 a column labelled "Bing (Copilot proxy)" read Bing's organic results as a stand-in for Copilot; we suspended it that day on finding it had been answering a different query than the one asked. Its rows stay labelled as a proxy, were always excluded from sentiment, position and your share against rivals, and are never counted as Copilot answers. API answers can differ from the consumer apps (no account memory, no custom instructions). Other tools collect differently — some read the public web interfaces directly — so we document our collector per surface here rather than claiming everyone measures alike.
02
How many times we ask
In plain words: the same question can get a different answer each time. So the weekly checks and the checks you start yourself ask the chat assistants twice and count you as named if either answer names you.
Answers to the very same question commonly vary 10–34% from one ask to the next, so any single answer is a snapshot, not the final word. To reduce that noise, the weekly checks and the checks you start yourself ask each AI chatbot (Perplexity, ChatGPT, Gemini, Claude, and Grok when an account adds it) more than once — two samples by default — and count you as mentioned if ANY sample mentions you, since missed mentions are where sampling noise hurts most. Google's AI Overviews and AI Mode, read from Google's results page, are checked once. The daily checks ask each AI assistant once, which is why a question is only called lost after two checks in a row miss it. Each result records how many samples agreed, and trends across checks are still the signal: a single flip can still be noise.
03
How often we check
How often we check depends on your plan:
- Free: a first check on all 5 default AI assistants, then one automatic check a month on Perplexity, with the same two samples per question, and the owner is emailed what changed since the previous check.
- Starter: a weekly automatic check on its AI assistants (by default ChatGPT, Perplexity and Gemini) plus a lighter trend check every 3 days on 10 questions you pick — a single ask per question that adds a point to your history and share-against-rivals charts. If a Starter account turns on daily checks for its low-cost assistants, those daily checks replace the trend check.
- Pro and Agency: every question is checked on all 5 default AI assistants, daily on Perplexity, AI Overviews and Google AI Mode, one ask per assistant, and weekly on ChatGPT and Claude, and on Gemini and Grok if you add them — the weekly deep check, two asks per assistant, which asks the daily assistants too on its day. A Pro or Agency account that turns daily off gets the trend check back, daily, on the questions you pick (10 on Pro, 30 on Agency). Your dashboard's latest results show each assistant's most recent check — today's for the daily assistants, the weekly deep check for ChatGPT and Claude — each with its own date.
- Checks you start yourself: you can run a check any time within your plan's fair-use allowance, and such a check covers all 5 default AI assistants on every paid plan unless you choose other assistants. Every check — automatic or started by you — runs on a durable background queue; checks start immediately, small runs often finish in about a minute, and larger ones update in the background with live progress.
You choose your AI assistants, and which of them run daily, in Settings; daily costs more questions than weekly, and fewer or cheaper assistants let you track more. Alerts when AI starts or stops naming you come from the daily and weekly checks alike, each compared only with earlier checks of the same kind: a question is called lost only after two checks of that kind in a row miss it, and never while the latest check of the other kind still mentions you. History and share-against-rivals trends are built from all data; the Visibility Score reads the latest results. The AI Source Index keeps its own fixed weekly question list, unchanged.
04
What counts as "mentioned" and "cited"
Cited = the AI linked to a page on your domain, and your subdomains count as you (on your dashboard: “Linked to your site”). Mentioned = cited, or your brand name / domain appears in the answer text as a whole word (“Named you”). Position = where you appear in the list of brands the answer names (1st, 2nd, 3rd…), saved when the check runs.
05
Why a "lost" alert takes two checks
A win alert is immediate. A loss alert only fires after you're missing from two consecutive checks — and never when an AI assistant merely errored — so one noisy answer can't email you "AI no longer cites you". This deliberately delays true losses by one check.
06
Four results we never mix up
“Check failed” (the AI service was down), “No AI answer” (e.g. Google showed no AI Overview), “Not configured” (we have no access to that AI assistant yet) and “Didn't name you” are four different results, each shown with its own label. A gap in the data is never shown as a loss.
07
What we don't do
No fabricated results — if there's no data, we say so. The Site check grade (A–D: how ready your pages are for AI to read and quote) is a weighted checklist of verifiable signals, not a ranking prediction: nobody can guarantee AI will recommend you, and we don't.
08
How we read sentiment
Paid plans: the answers that name you are read by a small model (Claude Haiku) at least weekly for each question and AI assistant within the plan's monthly allowance, newest first, and labelled positive, neutral or negative toward your brand; an answer is read at most once and its label is kept for good, answers past the month's allowance stay unread, and an answer that only repeats your name from the question is not counted. Free: a keyword estimate.
09
Every percentage shows how many answers it's based on
In plain words: next to every percentage we show how many answers it comes from, and we mark it when there are too few answers to trust it.
A percentage on its own hides how many answers it came from: 2 of 10 and 200 of 1,000 are both 20%. So every rate that is a count out of a count (share of voice, the questions an AI assistant names you on, the warning that answers describe a different business) is shown with the number it was measured on and a 95% Wilson score interval: on the dashboard, in the client report, in the CSV export and in the API, where each rate field travels with its counts and interval. The weekly summary email prints the counts themselves and marks a sample that is too small; the interval is on the dashboard. We use Wilson rather than the textbook formula because at the small samples this product produces it stays between 0% and 100% and does not collapse to a single point when none, or all, of the answers count. A rate measured with fewer than 32 answers is marked as too small to read as a rate: at that size its 95% interval can span more than a third of the scale. The rule reads the sample size alone, so two rates measured on the same number of answers are always marked alike. Share of voice counts mentions (each business at most once per question), and one answer can name many businesses, so those mentions are not independent: its sample size is the number of questions measured that don't include your name. Below 10 of them it is marked too small to read; from 10 up to 32 it is shown as an early reading, which firms up as more questions are measured; and it is always printed with that number beside the mentions (“Mentions counted: N · questions: Q”). Scores that are not proportions (the Visibility Score, the Site check score, an average position) carry no interval, and say so, because inventing one would be false precision. For the AI Source Index we publish every observation behind the weekly tables, row by row, so anyone can recount them.
10
How the Visibility Score is built
In plain words: the score mixes five things: how often AI names you (30%), your share against rivals (25%), how many AI assistants pick you up (20%), your Site check result (15%) and how kindly you are described (10%).
The AI Visibility Score blends five components, measured only on your questions that don't include your name: coverage, the share of your questions where an answer names you (30%); your share of voice among the businesses AI names, measured against the most-named one so that the leader scores in full (25%); how many of the AI assistants that answered pick you up (20%); your Site check score: its own checks, without its two mention items (whether AI names you, your share of voice), which the first two components already count (15%); and sentiment, how favorably the answers that name you describe you (10%). Sentiment is pulled toward neutral by the size of its sample: it is read as if 5 neutral answers were added to the ones that name you, so one positive answer counts as 58% rather than 100%, and the pull fades as answers accumulate. When no answer names you, sentiment counts as zero rather than dropping out: not being named is a measurement, and dropping it would hand its weight to the other components and lift the score of a brand nobody names. A component we could not measure, such as the Site check score before your first Site check or sentiment on answers not yet read, is left out and the remaining weights are rescaled. Until at least 5 of your questions have an answer that names you, every place that shows the score says how many do: on so few answers, a single one can move it by many points. The formula carries a version number in the API, so a stored score is never compared across a change like this one.
11
The searches the AI ran before answering
In plain words: before answering, an AI often searches the web itself, and not always with your exact question. When it tells us what it searched, we show those searches on each question's page.
A grounded AI assistant runs its own web searches before it answers, and they need not match the question you asked. Where the assistant's API reports them, we store those searches with every answer and show them on each question's page, in the API and over MCP: the distinct searches across assistants, which assistants ran each, and how many. We capture them from ChatGPT (the web_search calls in OpenAI's response), Claude (its web_search tool calls), Gemini (the search queries in its grounding metadata) and Perplexity (the queries its Agent API reports). For Grok we record them when xAI's response includes the search text, and nothing when it does not. Google AI Overviews and AI Mode do not expose them: what we read from Google's results page carries the AI's answer and sources, not the searches behind it, so nothing is shown for them. When an assistant searched and did not say for what, we record nothing rather than guess. Until a question has observed searches, a separate button can predict likely sub-questions with a small model; that prediction is labelled as one, never stored, and never counted as a search.
12
Did a fix work? The held-back comparison
In plain words: a week after you finish a fix, we compare that question before and after, and against similar questions we gave no advice on, so a change that happened everywhere is not counted as your win.
Every fix you finish on one question is judged on its own. Before is the newest full check that asked the question before you marked the fix done or published its draft; after is the first full check at least seven days later, counting only the AI assistants that answered both times. The verdict stays with that pair: later checks do not judge it again, because the same interval read every week would sooner or later clear zero on noise alone. The same two checks are read for the questions held back from advice, about one in five, picked by a fixed rule before any advice is written and frozen in each check's record, and counted one question at a time. The verdict is the difference between the two changes, with a Newcombe interval built from each side's Wilson interval: up or down only when the whole interval sits on one side of zero, no change when both changes stay within ten points, and too close to call otherwise. It needs at least four of each, assistants that answered both checks and held-back questions, so only checks that ask four or more assistants can judge a fix: Pro and Agency's default set does, Starter's does not, so a Starter account needs a fourth assistant picked. It refuses rather than guesses when the two checks grouped the questions differently. It is a measured difference, not proof of cause: assistants change on their own schedule and are not independent of each other. Changes to the whole site (llms.txt, schema, robots.txt) reach the held-back questions too, so they are not judged.
13
This method, running on us
Everything above also runs against Promvia itself, and the result is public — including while AI assistants rarely name us for the questions we track (the live scoreboard has the current number). A methodology page you cannot watch operating is a promise; this one has a scoreboard.
See Promvia's own live results →