CitadexDocs

AI Traffic — Crawlers & Pages, Agents and the Optimization Plan

The three tabs of AI Traffic: Overview (AI search fetches, fetch error rate, AI referral visits, To handle this week, fetches and referrals by AI platform, the trend), Crawlers & pages (To fix, AI crawlers, Page performance, AI fetches of new articles) and Agents; the Data from line under the tabs, and connecting Google Analytics, the tracking snippet, Cloudflare, Vercel, WordPress, Next.js or Netlify on Integrations; the Optimization plan opens from the page bar on every tab. Visits, orders and revenue per engine and landing page are on Revenue Attribution.

Visibility22 min read

What it measures

Citation data tells you what AI says. AI Traffic tells you what happens on your site: which AI crawlers read your pages and whether they hit errors, whether the pages they read are cited in AI answers, how many visitors arrive by clicking an AI answer, and whether AI agents can complete tasks on your site. Visits, orders and revenue per AI engine and landing page are on Revenue Attribution: the former AI referrals tab is now part of it, and old links open Revenue Attribution directly. The page shows only connected data and stays empty until a source is connected.

Tabs

TabWhat it shows
OverviewThree numbers (AI search fetches, Fetch error rate, AI referral visits), To handle this week, Fetches and referrals by AI platform, the trend; a chat box for the AI assistant at the bottom of the page
Crawlers & pagesTo fix, AI crawlers, Page performance, AI fetches of new articles
AgentsSite verification, agent readiness monitoring (with the full checklist) and simulated agent tasks

The dark Optimization plan button on the right of the page bar opens the plan from any tab; ⋯ next to it holds only Create llms.txt. Under the tabs is the Data from line: one chip per data source, green when connected, amber one step left when a connection was started but not finished, and grey (for example Cloudflare not connected) for the one most worth adding, chosen from your site. Click any chip to open its connection dialog right on the page; Manage connections goes to Integrations. Its ⓘ explains the evidence levels; each number's own level is in its ⓘ. On the right are Last 30 days (7 / 30 / 90 days) and translate. Without data, the Overview numbers show "—" and say which data is missing (for example No visit data connected); on Crawlers & pages, the AI crawlers block says what connecting gives you, with a Connect … button that opens the same dialog and Other ways (other connection options on Integrations).

Overview

Three numbers, with the change since the previous period beside each and how it is counted in its ⓘ:

  • AI search fetches: fetches by AI search and user-triggered crawlers. Model-training crawlers (such as GPTBot and ClaudeBot) are not included. The line below names the main AIs.
  • Fetch error rate: the share of fetches that returned a 4xx or 5xx status, counting content pages and AI search and user-triggered crawlers only. The line below says "N pages still return errors".
  • AI referral visits: visits that arrived by clicking a link in an AI answer. View attributed revenue → opens Revenue Attribution with the orders and revenue these visits brought.

To handle this week lists what needs attention, in order; the first item's button is dark:

  • robots.txt disallows an AI search or user-triggered crawler: that AI cannot fetch your site, so it will not cite it in answers. How to fix goes to To fix on Crawlers & pages.
  • Pages that still return errors to AI crawlers (See pages). Once a page is fixed and AI crawlers fetch it again without an error, the item disappears.
  • The pages the optimization plan recommends updating first (Optimization plan; open Show N pages for See suggestions on each page). This week's tasks on the Dashboard read the same plan.
  • New articles that no AI search crawler has fetched 3 days after publishing (Submit to IndexNow).

When groups of pages target the same prompts, a last line suggests merging them or assigning each prompt to one page; Ask the AI assistant has the assistant propose which page should own which prompt. With nothing to handle you see "Nothing to handle: AI crawlers can read your site."

Fetches and referrals by AI platform counts only crawlers that fetch to answer (search and user-triggered). One row per platform: answer fetches, AI referral visits, referral visits per 100 fetches and the assessment (Strong referral rate / Low referral rate / No referral visits yet / Model training only). The line under the table names the platforms with the most referral visits; crawlers used only for model training send no referral visits, and you can disallow them in robots.txt if needed without affecting citations in AI answers.

Trend shows answer fetches (left axis) and AI referral visits (right axis) by day.

The chat box at the bottom of the page ("Ask about the reasons behind these numbers or what to do next…") offers up to three ready questions built from the page's numbers, for example "Which pages are fetched by AI search crawlers but not cited, and which should I update first?". One click sends it to the AI assistant, which answers with this project's data; you can also type your own question.

Crawlers & pages

AI companies run three kinds of crawler, for different purposes:

PurposeWhat it doesExamples
SearchAI search indexing: decides whether AI answers can find and cite youOAI-SearchBot, PerplexityBot, Claude-SearchBot, Applebot
User-triggeredA person asked the AI to open your pageChatGPT-User, Perplexity-User, Claude-User
TrainingUsed to train future modelsGPTBot, ClaudeBot, Bytespider, CCBot

Disallowing search or user-triggered crawlers directly affects whether AI can cite you now; whether to allow training is your call.

AI Traffic → Crawlers & pages

AI Traffic → Crawlers & pages

From top to bottom, the tab has:

  • To fix. When robots.txt disallows an AI search or user-triggered crawler, the rules to add to robots.txt are here (Copy lines); changes usually take effect within a day. Below them are the pages that still return errors to AI crawlers: the status code and what it means (such as Page not found, or Access denied (firewall or security settings)), the count and the crawlers; Copy for your developer produces a note you can send as is. Only pages whose latest fetch still failed are listed, so a fixed page drops out.
  • AI crawlers. One row per crawler: purpose, fetches (last N days), status and most-fetched page ("N pages ▾" opens the pages it fetched). Each crawler has one status: Disallowed by robots.txt; Simulated request blocked (Citadex requested your home page with this crawler's user agent and a CDN or firewall blocked it; hover for details. The request comes from Citadex's servers, so if your firewall only lets official crawler IPs through, a block here is expected); Allowed (model training); Can fetch. Beside the status are "N verified" (visits whose IP passed the check) and "N spoofed requests excluded" (claims to be an AI crawler but the IP is not the operator's; never counted in any number). Check now at the top right runs the simulated request again, with the date of the last check beside it.
  • Page performance. One row for every page AI search crawlers fetched or AI answers cited: AI fetches, AI citations, Google clicks (once Search Console is connected) and AI referral visits. AI citations are answers to your tracked prompts that link to the page, counted once per answer; AI fetches are successful fetches by AI search and user-triggered crawlers. The filters are All, Fetched, not cited (AI has read the page but not cited it yet; the row has See suggestions) and Cited, not fetched recently (AI answers may still rely on an old copy), plus, with Search Console connected, Strong on Google, weak in AI and Strong in AI, weak on Google to find pages that perform very differently on Google and in AI.
  • AI fetches of new articles. Articles published to your site through Citadex in the last 60 days: published date, first AI fetch (days after publishing, and which crawler), fetch count and the prompts that cite it. An article must be fetched before AI can cite it. For one not fetched yet, Submit to IndexNow submits its address to search engines that support IndexNow; it uses the key set up in Integrations → IndexNow and asks you to set one up first when there is none.
  • Click any page to open its details: AI fetches, AI citations, AI referral visits, errors and the prompts that cite it; Open page opens it on your site.

Optimization plan

The Optimization plan (the dark button on the right of the page bar, on any tab; See suggestions opens one page's change list directly; link /ai-traffic?plan=1) lines up the pages to change, in two tabs:

  • This week: the pages most worth fixing first. Each shows why (for example "Fetched N times by AI · cited 0 times · N prompts", or cited but bringing no AI visits), with a Change list button, or Full rewrite for a page too thin for a few edits to work. Below them you can open More (N pages), Changed lately, being watched (N) and Done, being retested (N) (with View retest and Reopen). You don't need to tell Citadex when you change a page: it reads the pages it would list again, and when the text changed (or the page declares a newer update date, or Citadex published it) the page moves to Changed lately, being watched, with the day it changed, how long it is watched and its AI reads and citations since. While it is watched it is not listed, and the AI assistant and the weekly plan don't repeat the same advice. After the watch, only the numbers since the change decide whether it comes back; its change list is made again from the new page. When groups of pages target the same prompts, a line says so here too, with Ask the AI assistant.
  • Change list: the specific edits for one page: now → change to, and additions, each marked Important, Suggested or Optional; Q&As to add come with the structured-data code (Copy Q&As, Copy structured-data code). Change lists don't count as an article; Full rewrite writes a new version of the whole page and counts as one article. When your site is connected through WordPress, Ghost, the Citadex Shopify app, HubSpot (blog posts), Webflow (CMS items), Wix (blog posts) or GitHub (a static site), Write to site opens a dialog listing each edit as now → change to, where you can leave out any edit, the FAQ or the whole page; once you confirm, Citadex writes the edits to the page itself (title and address stay; the previous content is kept and can be restored page by page or all at once) and a retest starts. HubSpot, Webflow and Wix pages can be kept as a draft (Save as a draft): the edits wait in that CMS until you publish them there. On GitHub the edits become a pull request and take effect once it is merged. A page counts as done, and its retest starts, once the live page shows the new text; with an IndexNow key on the project, its address is also submitted. For pages AI cites less whose change list can be written, the AI Assistant suggests writing them in the chat. For HubSpot, click Integration on the HubSpot card on the Integrations page and enter a private app access token with the content scope. After editing by hand, click Done, start retest. The AI-only version holds only sentences already on the page, so AI crawlers and visitors get the same content.

Which page owns each prompt is assigned automatically; the top of the change list names the prompts the page owns. The AI Assistant's weekly plan prepares change lists for up to 3 pages in advance; see Weekly plan.

AI-optimized versions

When you can't edit the site, create an AI-optimized version of a page: only AI crawlers see it, while visitors, Google and Bing keep getting the original. It only uses what the original says, rearranged so AI reads and cites it more easily; numbers, names and claims the original does not have are removed. When the original changes, the version is rebuilt and switched off until you check it. Preview compares the two; once AI only is on, the table shows reads and citations since switching on. It needs AI crawlers connected on Integrations through the Cloudflare Worker, Next.js, Netlify or the WordPress crawler plugin.

Agents

The Agents tab (paid plans) starts with site verification. It checks your site's agent readiness every Monday and keeps a history chart, new issues and fixed items, and the error rate of search and user-triggered crawlers; it alerts you when the score drops 10 points or more or an item starts failing. Check now runs a check right away; Full checklist (N items) opens every item right on the page, issues first, each with its fix (Hide checklist closes it). Simulated agent tasks let a read-only AI agent try tasks on your site such as finding a price, a product, the refund policy or an inquiry form, and record whether it succeeded, the step it could not complete and why. See Agent readiness.

Two of its items look at key pages the way AI crawlers read them (no JavaScript). Key pages reads the pricing page (found in the sitemap or at /pricing), service and product pages: a page that returns only placeholder content such as “Loading…” fails, and a pricing page with no readable price gets a warning. Prices agree compares the prices in the structured data (JSON-LD), the prices visible on the page and the official prices in Brand info, and warns with the numbers when they disagree. When AI says your pricing is unclear or quotes an old price, start here: put the prices in server-rendered HTML, make the structured data match the page, fill in the current prices in Brand info, then update old prices on review sites.

Why verify site ownership? Continuous monitoring, simulated agent tasks and Cloudflare logs visit or read your site repeatedly, so they are only available for verified sites (nobody can point them at someone else’s site). At the top of AI Traffic → Agents, pick one method: a DNS TXT record on the domain, a meta tag in the homepage <head>, or a file at /.well-known/citadex-verification.txt, then click Verify. Connecting Cloudflare with a zone for the project’s domain verifies it automatically. Verified sites are re-checked monthly; two misses in a row remove the verification. Existing projects keep their weekly automatic checks during a grace period; Check now and Run need verification.

What are the Confirmed / Platform-reported / Self-reported / Estimated / Unknown tags next to numbers? The evidence level: where a number comes from and how far to trust it. Confirmed: your own data or orders that can be checked one by one (form submits seen by the site script, AI-chat orders on a channel the merchant confirmed). Platform-reported: numbers from another platform, passed on as reported (Cloudflare, Search Console, GA4, Shopify channels, the CRM’s original-source field). Self-reported: what the buyer said (the CRM’s “how did you hear about us”, manual links). Estimated: computed by a method, named on hover (probabilistic visitor–contact match, same-company contact, modeled from AI session share, multi-touch split, difference-in-differences against control pages, crawlers identified by user agent only). Unknown: the source was not recorded. When a total mixes sources the tag shows the weakest level marked “mixed”; hover for the share per level. Every revenue attribution tab, AI traffic, demand in prompt coverage, incrementality experiments, results reports and CSV exports carry it; attribution records can be filtered by it; MCP returns evidence_level.

What are fix approvals? Every fix Citadex generates (change lists in AI traffic → Optimization plan, page rewrites, articles written from This week’s tasks / the AI assistant / MCP, market and citation-source repair plan tasks) runs only after approval: marking a page fixed, publishing or marking a task handled needs the owner, an admin or the client to approve; members can request approval. Articles and page rewrites are the exception: in a project without client accounts (not a white-label client project) they are approved automatically for the owner when generated (recorded as an automatic approval) and can be published straight away; projects with client accounts keep the approval step. An approval covers the content fingerprint it was given, so when the content changes (a rebuilt list, an edited article) the old approval no longer applies and it needs approving again. When an owner or admin clicks publish or mark fixed on something not yet approved, “Approve and run” appears and both steps are recorded. Rejecting needs a reason. Dashboard → Fix approvals filters by state, shows what each fix changes and who approved / rejected / ran it and when, and exports the audit log (CSV). Agencies can approve for their client (recorded as “Agency (for the client)”) or create a client approval link: the client reviews, approves or rejects without signing in; links last 14 days and can be revoked; add the client’s email to send it under the agency’s brand. The MCP tool get_fix_approvals reads the queue.

How does the fact check in Brand fact check work? In Automations → Brand fact check → Official facts → More official facts (optional), list the brand’s official facts (founding year, headquarters, product lines, price tiers, certifications, regions served, contact details; each can apply to one language only). Every day AI reads the answers that mention you and compares each statement about the brand with the list: consistent, contradiction, or not covered. Contradictions are listed under Problems found, ranked by severity (high: price, contact, certifications; medium: products, regions; low: founding year, headquarters and other history), each with the AI’s words, the sources the answer cited, and how to fix it: your own cited page → edit that page; a third-party site → ask them to correct it; no source → state it clearly on your site. High and medium ones go to This week’s tasks and the daily alerts (with a cooldown, so the same problem isn’t repeated). Check now runs once every 10 minutes (up to 20 answers). Changing the list re-checks answers. The prices typed in Official facts count as official prices too; no need to repeat them in the list. The check reads the AI answers saved at each scan (not live), and each statement shows when it was last seen. When newer answers to the same prompt on the same engine no longer say it, or it came from your own page and that page was changed since, it is greyed out with the reason, not counted, and kept out of This week’s tasks and the alerts. The MCP tool check_brand_integrity returns the contradictions.

How is the data collected and judged? Is it documented publicly? Yes. The Data & methods part of the public methodology page /ai-search-methodology (English at /ai-search-methodology/en, Japanese at /ai-search-methodology/ja) explains how each AI engine is captured today (real product UI, official API + web search, our market search then the model, Google results page), the models scans actually used in the last 30 days, how the region is applied (the search runs from the market’s country, or the country is named in the request), sampling (once per engine per scan, 3 times with repeated sampling), the five recommendation labels with the bar for switching a language to the model judge and the languages already switched, the five evidence levels, the 7-day ChatGPT data-source comparison and the known limitations. It reads the configuration and data the product runs on, refreshes every 6 hours, and says the same thing the product does.

Are the AI crawler visits real? What do Verified / Unverifiable / Spoofed mean? Anyone can set their user agent to GPTBot, so every crawler visit is checked the way the crawler’s operator publishes: OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User) and Perplexity by their official IP ranges; Google, Bing and Apple by IP range or reverse DNS; Amazon by reverse DNS (the name must be the operator’s and resolve back to the same IP). Verified: passed, evidence level Confirmed. Spoofed: claims to be an AI crawler but the IP is not the operator’s; kept apart and never counted in visits, trends or alerts. Unverifiable: the log has no IP, the crawler publishes no way to check (e.g. ClaudeBot, Bytespider) or the lookup timed out; counted on the user agent. To get IPs into your logs: the direct Cloudflare connection includes them; the Worker, Next.js, Netlify and WordPress forwarding code now sends the IP too, so copy and deploy it again if you installed an older version. On AI traffic → Crawlers & pages, the AI crawlers table shows one status per crawler: Disallowed by robots.txt when robots.txt does not allow it; Simulated request blocked when the simulated check is blocked (we request your home page with each crawler’s user agent to see whether the CDN or firewall blocks it; the request comes from Citadex’s servers, so if your firewall only lets official IPs through a block there is expected; Check now runs it again); Allowed (model training) for training crawlers; Can fetch otherwise. Beside the status is what the real access logs show: "N verified" and "N spoofed requests excluded". The AI Crawler Access Checker tool also shows the declared and simulated results.

How do I connect Cloudflare? Integrations → Cloudflare (or click Cloudflare in the Data from line on AI Traffic): click Authorize with Cloudflare, approve read-only access on Cloudflare and pick the domain when you come back, or keep pasting a read-only API token (Zone · Analytics · Read) and the Zone ID. Disconnecting an authorized connection also revokes the authorization at Cloudflare. Enterprise domains can turn on Full logs (Logpush): Cloudflare pushes every AI-crawler request (with IP), more complete than the hourly summary, which stops while it is on; with an API token the token also needs Zone · Logs · Edit.

AI Traffic → Agents

AI Traffic → Agents

Connecting data on Integrations

Every AI Traffic source is connected on Integrations in the sidebar. Each source has a card, and every card opens the same centered dialog: on the left, what it reads and where it is used; on the right, its own steps and settings. Suggested for you (under Suggested) lists the ones that fit your site best, for example "Your site runs on Cloudflare". To stay on AI Traffic, click a chip in the Data from line, or Connect … in an empty block: it opens the same dialog.

Each project has its own ingest key; Use a new key makes the old one stop working at once. Only the workspace owner can view the key, with Show key, and each view is recorded; members can create or replace the key but cannot view it, and connect through Cloudflare or the WordPress plugin instead. The receiving addresses on the revenue attribution cards work the same way: only the owner can view them, with Show address. The Zapier / webhook publishing address shows only that it is saved and its host; only the owner can view the full address, with Show full address. Pick one or more:

  • Google (Google Analytics conversions): Integrations → Google, click Connect Google, then choose the property. No code; the same authorization also covers Search Console. Only the workspace owner can connect it.
  • Site tracking snippet: Integrations → Site tracking snippet. Paste one line into your site's head to record visitors who arrive from AI; call the conversion function on your goal to record conversions. Use it for sites without Google Analytics; revenue attribution relies on it too.
  • AI crawler monitoring (server logs):
  • Integrations → Cloudflare: Direct connection (recommended) takes a read-only API token and Zone ID and pulls AI-crawler requests every hour, with no Worker to deploy; a Worker is also available.
  • Integrations → Vercel: Team settings → Drains, add a log drain with the address shown in the dialog as the destination.
  • Integrations → WordPress crawler plugin: download the plugin, upload and activate it.
  • Integrations → Next.js · Netlify · other servers: for Next.js or Netlify, save the code shown in the dialog in your project and redeploy; for another server, forward AI-crawler requests following the example in the dialog.

Even with nothing connected, To fix checks your robots.txt and lists the AI crawlers it disallows with the rules to add; the robots.txt check card on Integrations runs the same check on its own. To continue the funnel to leads, deals and revenue, use Revenue Attribution; see Revenue attribution.

Alerts

Turn on "AI traffic issues (crawl drops, blocks, errors)" under Settings → Email Notifications → Project alert notifications to get an email or Slack message at your chosen frequency.

Frequently asked questions

Pages are cited by AI, so why no visits? Most AI answers carry no link, or people don't click. Check Fetches and referrals by AI platform on Overview: the more referral visits per 100 fetches, the more often that platform refers readers to your site. Orders and revenue per platform are on Revenue Attribution.

Where is the former AI referrals tab? Visits, orders and revenue per AI engine and landing page are all on Revenue Attribution, and old links open it directly. The AI Traffic Overview still shows the total AI referral visits; View attributed revenue → takes you to Revenue Attribution.

GPTBot fetched many pages, so why aren't they "Fetched, not cited"? Training crawlers such as GPTBot, ClaudeBot, Bytespider and CCBot collect data to train models, not to write answers. A page counts as read by AI only when an AI search or user-triggered crawler fetched it successfully, and AI search fetches and Fetches and referrals by AI platform count only those two kinds too.

Is this related to Search Console? AI crawler fetches come from your server logs, not from Search Console. With Search Console connected, Page performance adds a Google clicks column and the Strong on Google, weak in AI and Strong in AI, weak on Google filters. Search Console is also used for indexing pings after publishing and for importing prompts.

Was this article helpful?