Where the data comes from, how it is judged, and where it falls short

Data sources

Every prompt is asked in the market you choose, in that market’s country and language. The table shows how each engine is captured today and the models scans actually used in the last 30 days.

Real product UI

Asked in the engine’s consumer product: the answer and sources a user sees.

Official API + web search

The vendor’s model API with its own web search turned on. A benchmark; it may differ from what users see in the app.

Our market search, then the model

The engine’s API has no web search, so we search Google (Baidu for Chinese markets) from the market’s country and the model answers from those results only.

Google results page

Google’s AI Overview / AI Mode on the results page, fetched for the market’s country.

• Sampling: on each scan, every prompt is asked once per engine. Business and Agency plans can switch up to 20 prompts to repeated sampling: asked 3 times per engine on every scan, shown as x/3 with a 95% range.
• A failure is not a zero: an error, an exhausted quota or a timeout is recorded as a failure and retried, never as “not mentioned”.
• Raw answers are archived: the answer text, cited sources, capture path, model and the searches the engine ran before answering (where it exposes them), all visible in the evidence panel.
• Region and language: prompts are asked in the market’s language; engines that take a location search from the market’s country, the others are told the country in the request.

Recommendation judgment

Being mentioned is not being recommended. Every answer that names the brand gets one of five labels; no mention, no label.

1. Top pickNamed as the single best or first choice, with no condition.
2. RecommendedActively recommended to the person asking, with no restricting condition. One of several recommended picks counts here.
3. ConditionalRecommended only under a condition (“if your budget allows”, “for teams that need X”). Counted separately, not as a recommendation.
4. Listed optionListed among options with a neutral description, no recommendation.
5. Mentioned onlyNamed in passing, used as a comparison point, criticised or advised against.

• Being first in a list does not make a brand the top pick; the answer has to say so (“best”, “first choice”).
• Criticism or negation (“expensive and slow”, “I’d avoid it”) is Mentioned only, even when the brand is listed first.
• An AI model judges each answer and keeps the quote it relied on; code then checks that the quote really is in the answer.
• A language switches to the model only after its human-labelled set reaches accuracy ≥ 90% and F1 ≥ 0.85 for every label; until then keyword rules decide. Both carry a version, the switch-over day is marked on trend charts and earlier data is not recomputed.
• Users can correct a label in the evidence panel; the correction is stored beside the answer and the original answer is never changed.

All languages still use keyword rules for now (rec-judge-1); each switches once its labelled set passes.

Evidence levels

Every number in revenue attribution, AI traffic, demand, incrementality tests and results reports carries an evidence level: where it came from and how far to trust it.

1. ConfirmedFrom your own data or orders that can be checked one by one.
2. Platform-reportedReported by another platform (Cloudflare, Search Console, GA4, Shopify, your CRM); passed on as reported.
3. Self-reportedWhat the buyer said, e.g. their answer to “how did you hear about us?”.
4. EstimatedComputed by a method, not recorded one by one.
5. UnknownThe source was not recorded.

When a total mixes sources, it shows the weakest level marked “mixed”, with the share of each level. CSV exports and the MCP server return the same levels.

“Estimated” always names its method: probabilistic visitor–contact match; same-company contact stands in; modeled from AI session share; multi-touch split; difference-in-differences vs control; user agent only; search volume model; share of repeated samples

7-day ChatGPT data-source comparison

To check that the capture path itself does not change the conclusions, we asked ChatGPT the same prompts on the same day through our current pipeline and through a second vendor (Oxylabs), for 7 days, and compared the results.

The comparison is being compiled; results appear here once a full 7 days are published.

Known limitations

• AI answers vary from run to run. A single sample is noisy; trends are more reliable than one scan, and repeated sampling is recommended for important prompts.
• We ask signed out, with no personal history or memory. Signed-in users may see personalised answers.
• Engines captured through the official API are a benchmark and may differ from the app; the table shows each engine’s capture path.
• Some engines take no location, so the country is only named in the request; that is weaker than a search that truly starts in the country.
• Tracked prompts are a representative set you choose, not every real question; demand figures are estimates.
• The model judge scores 90% or more on the labelled sets but still makes mistakes; keyword rules err more on complex sentences.
• Engines expose only the citations and searches they choose to; where an engine shows no searches it is marked “unknown”.
• Crawlers identified by user agent alone may be spoofed, so they are marked Estimated; many AI apps open links without a referrer, so AI-referred visits are undercounted.