Agent Readiness: AI Crawlers, the Readiness Score and Simulated Agent Tasks
How AI crawlers are split by purpose (search, user-triggered, training) and what each verdict means, the agent readiness score, the free public check and AI crawler checker, and weekly monitoring and simulated agent tasks on AI Traffic → Agents.
Three kinds of AI crawler
AI companies send crawlers for three purposes, and each needs different handling. On AI Traffic → Crawlers & pages, the Purpose column of the AI crawlers table says which kind each crawler is:
| Purpose | What it does | Recommended |
|---|---|---|
| Search | AI search indexing: decides whether AI answers can find and cite you | Allow |
| User-triggered | A person asked the AI to open your page | Allow |
| Training | Used to train future models | Your choice |
With crawler logs connected, the table has one row per crawler with its fetches over the last N days, its most-fetched page and one status:
- Disallowed by robots.txt: your robots.txt does not let it fetch.
- Simulated request blocked: a request for your home page with this crawler's user agent was blocked by a CDN or firewall. The request comes from Citadex's servers, so if your firewall only lets official crawler IPs through, a block here is expected. Check now runs the simulated request again.
- Allowed (model training): a training crawler; allowing it is your choice and doesn't affect citations in AI search.
- Can fetch: nothing in the way.
Beside the status, "N verified" counts real visits whose IP passed the check, and "N spoofed requests excluded" counts requests that claim to be an AI crawler from an IP that is not the operator's; they are never counted in any number. When a search or user-triggered crawler is disallowed or blocked, AI cannot fetch your pages, so it cannot cite you.
When robots.txt disallows AI search or user-triggered crawlers, To fix on the same tab lists the rules to add to robots.txt; Copy lines copies them.

AI Traffic → Crawlers & pages
The agent readiness score
Whether AI agents (AI that finds information, compares and buys on a person's behalf) can find, read and operate your site and complete a purchase, scored 1–100 in four groups: Discoverable, Readable, Operable and Transactable. The checks cover robots.txt access for AI search and user-triggered crawlers, a sitemap, llms.txt, main text readable without JavaScript, structured data, page structure tags, form field labels, button names, login walls, and on product pages Product structured data, price, stock and a buy button. On sites that are not stores, the store items show Not applicable. llms.txt is optional: Google says it does not use it and no other AI engine has publicly confirmed reading it, so a missing llms.txt is not an issue. It stays out of the fix list and change alerts.
Free tools: the public check and the AI crawler checker
- Agent Readiness Check (no login needed): enter a site and click Check. Signed out you see the score, the four groups and Main issues; after Sign up free you see the full checklist with a fix for each item, plus Copy fix list and Ask AI Assistant how to fix.
- AI Crawler Access Checker under Tools in the sidebar: which AI crawlers your robots.txt lets in, by purpose, each Allowed, Partly blocked, Blocked or Could not read robots.txt; a blocked or partly blocked search or user-triggered purpose is shown in red.

Agent Readiness Check result
Agents tab: weekly monitoring (paid plans)
AI Traffic → Agents, available on paid plans.
The Agent readiness card checks your project's site automatically every Monday and shows the score (last time's in brackets), the four group scores and a history chart. Below it are the New issue and Fixed items since the last check, with the fix for each new issue, plus a line when search and user-triggered AI crawler visits get errors. If the score drops 10 points or more, or an item starts failing, you get a project alert. Check now runs a check immediately at no charge; Full checklist (N items) opens every item right on the page, issues first, each with its fix (Hide checklist closes it).
Simulated agent tasks
Simulated agent tasks sends a read-only AI agent to try real tasks on your site.
What the agent can and can't do: it only opens and reads pages of your site, at most 8 pages per task. It does not run JavaScript, log in, submit forms, buy or book anything.
Default tasks: six, chosen by site type.
- Store: Find the price, Find a specific product page, Check stock, Find the refund policy, Check shipping to {main market country}, Find a way to contact them
- Service: Find the price, Check service in {main market country}, Find the inquiry form and its required fields, Find where to book a demo, Find the refund policy, Find a way to contact them
Edit tasks lets you remove tasks or type your own (for example "find the price of the enterprise plan") and click Add, up to 10, then Save.
Reading a result: one row per task.
- Result: Done or Failed.
- Where it got stuck: the answer it found when done; "Step N: …" when failed, the step and the element that blocked it.
- Time: steps and seconds.
- Steps: expand to see each page it opened and what it saw.
Running: the tasks run automatically once a week at no charge. To run now, click Run; you first see how many credits the run uses, and it starts only after you click Run again to confirm. If credits are short you see Not enough credits.

AI Traffic → Agents
FAQ
Should I allow training crawlers? They do not affect whether AI search can cite you, so it is your call. Allow search and user-triggered crawlers.
The Agents tab says "Add the project’s website first". Set the website in the project settings; checks and tasks need it.
A task failed. What do I change? Look at Where it got stuck and Steps. Common causes: price or stock loaded only by JavaScript, or information behind a login. Apply the fixes from the readiness checklist; the next weekly run re-checks automatically.
Is any of this free? The public check and the AI crawler checker are free. Weekly monitoring and simulated tasks on the Agents tab need a paid plan.
Was this article helpful?
