AI Crawler Audit: What 118 AEO Vendors Reveal About robots.txt and llms.txt
A 118-site AEO vendor sample suggests that AI-crawler access is usually open, while llms.txt adoption is common enough to test but not proven to improve AI citations.
· 13 min read
In a check of 118 AEO vendor domains, 116 served a robots.txt file, 64 served an llms.txt file, and no AI crawler tested was root-blocked by more than one site. That result makes a useful starting point for an AI crawler audit: it can reveal access and configuration quickly, but it cannot by itself prove that a page will be crawled, retrieved, or cited in an AI answer.
The Reddit analysis that prompted this discussion examined companies selling AI visibility services, tools, or agency work. Its headline result is less a best-practice benchmark than a reminder to separate three questions that are often conflated: whether a bot is permitted to crawl, whether it can successfully fetch a page, and whether an answer engine finds the page useful enough to use. A practical audit needs evidence for all three.
The 118-site result: open access is the default
The source analysis sampled 118 AEO vendor sites and requested two root-level files from each domain: /robots.txt and /llms.txt. It reported that 116 of 118 domains returned a robots file, while 64 of 118 returned an llms file that met its validity test: an HTTP 200 response beginning with a Markdown H1.
The audit checked 14 named AI-related user agents and tokens, spanning OpenAI, Anthropic, Perplexity, Google, Apple, Common Crawl, ByteDance, Meta, and Amazon. According to the published summary, no individual crawler was blocked at the site root by more than one domain.
That is a notable operational finding. Firms that sell AI visibility are not broadly using root-level Disallow: / rules against the AI agents in the test. But it should not be read as proof that every one of those sites is visible in ChatGPT, Perplexity, Gemini, Claude, or another product.
A crawler can be allowed in robots.txt and still fail to collect useful content because of:
- a CDN, WAF, bot-management challenge, or rate limit;
noindex, authentication, geo restrictions, or JavaScript-only rendering;- a weak internal-link path to important pages;
- thin, duplicated, outdated, or poorly structured content;
- a bot identity that differs from the one a site owner expected to see.
That distinction is central to an AI crawler audit. Configuration is a necessary diagnostic layer, not a visibility result.
Why robots.txt remains the first AI crawler audit check
robots.txt is the web’s crawler-preference mechanism. RFC 9309 defines a file of user-agent groups and rules that tell conforming crawlers which URI paths they may access. It is not an access-control system: a rule can communicate a preference, but it does not technically prevent an HTTP client from requesting a URL.
For website teams, the practical implication is clear. A root block such as the following is a strong, obvious signal that must be investigated:
User-agent: OAI-SearchBot
Disallow: /
Yet rules are frequently more complicated than that. An AI crawler audit should evaluate the relevant group, the target path, and any conflicting Allow rule rather than merely searching a file for a bot name.
Specific groups matter more than the wildcard group
Consider this configuration:
User-agent: *
Disallow: /private/
User-agent: OAI-SearchBot
Allow: /
A simplistic checker could report that the site blocks part of its content to crawlers generally. A rules-aware check must instead identify that OAI-SearchBot has a dedicated group and is allowed to fetch the whole site under this example.
RFC 9309 also describes matching behavior for user-agent groups and path rules. In a real audit, the output should state the exact URL tested and the rule that won, not merely label a bot “allowed” or “blocked.” This is particularly useful for agencies, where a single /clients/, /resources/, or /blog/ exclusion can matter more than the root setting.
Separate search, training, and user-triggered access
Not all bots from the same company have the same purpose. OpenAI documents separate controls for OAI-SearchBot and GPTBot, for example. Its documentation says a publisher can allow search crawling while disallowing use for generative-model training. That makes a blanket policy such as “allow OpenAI” too vague for a defensible audit.
The same principle applies across vendors. Teams should document each relevant agent by intended role—search retrieval, training, user-requested fetches, product validation, or another use—then decide policy deliberately. OpenAI crawler rules in robots.txt are best reviewed as individual controls rather than as one all-or-nothing switch.
An allowed crawler is not evidence of AI visibility
The Reddit sample found broad permission, but permission is not a citation metric. A page can be crawlable and remain absent from answers because the answer engine relies on different sources, sees insufficient evidence, cannot parse the content reliably, or decides a different page better serves the query.
This distinction matters because an AI answer engine is not simply a conventional search index with a chatbot interface. It may combine web retrieval, indexed data, maps, commerce feeds, model knowledge, citations from third parties, and query-specific ranking systems. Each provider can make different choices.
A useful evidence ladder looks like this:
- Policy evidence: the relevant crawler is not disallowed in
robots.txt. - Fetch evidence: server logs show verified requests that return successful responses.
- Content evidence: the fetched page contains clear, indexable, useful information.
- Answer evidence: testing shows the brand or page appears for relevant prompts.
- Business evidence: referred sessions, leads, or conversions are attributable where measurement allows.
Only the first layer was broadly measured in the 118-domain test. That is still valuable, because a hard block can invalidate later work. But the later layers determine whether AEO efforts produce practical visibility.
This is also why AEO and SEO should be treated as related but distinct disciplines. Technical crawlability, indexing foundations, content quality, entity clarity, and external credibility overlap substantially, while answer-engine appearance remains a separate outcome to observe rather than assume.
llms.txt: useful convention, unproven visibility lever
The source analysis found llms.txt on 64 of 118 sites, or roughly 54%. That adoption rate is significant enough to justify checking for the file, especially on developer platforms, documentation-heavy products, and sites where a concise curated content map may help downstream tools.
However, the file should not be marketed as a guaranteed AI citation mechanism. The llms.txt specification is a proposal for a Markdown-formatted file that can provide background information and links to important material. It recommends placement at /llms.txt or an appropriate subpath, but it is not equivalent to an official web-crawling control standard such as robots exclusion rules.
The original poster explicitly noted that they could not find evidence showing that llms.txt changes citations. That restraint is appropriate. A file’s existence tells an auditor that a site has published a machine-readable content guide; it does not demonstrate that a named answer engine requests, follows, or prioritizes it.
What a useful llms.txt can contain
For a site that chooses to publish one, the value should come from clarity rather than keyword repetition. A practical file can include:
- a concise description of the organization or product;
- links to canonical product, pricing, documentation, policy, and contact pages;
- a short explanation of which pages are authoritative for specific subjects;
- direct links to Markdown or clean HTML documentation where available;
- guidance that avoids conflicting claims between the file and the public website.
A poor implementation creates another maintenance surface. If an llms file lists retired pricing, old product names, or non-canonical URLs, it can confuse tools and reviewers even if no major answer engine uses it directly.
The right conclusion is measured: publish and maintain llms.txt when it is inexpensive and useful as a curated content artifact, but do not treat it as a substitute for accessible pages, sound technical SEO, accurate structured information, or actual answer-engine testing.
How to audit AI crawler access correctly
A robust AI crawler audit should begin with repeatable tests, not a manually copied robots file. The source material used one GET request per file, a 12-second timeout, and an RFC 9309-oriented robots parser. Those methodological details matter because a 200 response alone does not prove a usable file, and a timeout, redirect, soft error, or invalid response should be captured distinctly.
Use a layered crawl-access checklist
For every important host—such as www, root, help center, docs subdomain, ecommerce store, and staging environment—check the following:
- Fetch
/robots.txtand record final URL, HTTP status, response body, redirects, and response time. - Parse rules for each relevant AI user agent or permission token.
- Test representative URLs, including the homepage, a product or service page, a resource page, and a page intended to stay private.
- Fetch
/llms.txtif present and validate that it is useful Markdown rather than an HTML error page with a 200 status. - Check HTTP headers, meta robots directives, canonicals, and
X-Robots-Tagheaders on target pages. - Verify that CDN, WAF, bot protection, and server logs do not contradict the apparent robots policy.
An agency should avoid reducing the result to a single traffic-light score. “Allowed by robots, but no verified requests in the available 30-day logs” is much more informative than a generic green checkmark.
Audra is designed for this broader audit mindset: local-first crawling can combine technical URL checks with content and AI-answer visibility testing without making a file-level directive stand in for the whole result.
Server logs settle questions that files cannot
The most valuable next step after a robots review is log analysis. A robots file expresses a policy. Logs can show whether a known crawler actually made requests, which URLs it requested, the HTTP status returned, the user agent sent, and whether traffic was stopped upstream.
This is where vendor verification becomes important. A request that claims to be GPTBot, ClaudeBot, or another well-known bot in its user-agent string is not necessarily genuine. Where a provider publishes IP ranges or reverse-DNS verification instructions, teams should apply them before drawing conclusions.
A practical log review for the prior 30 to 90 days should answer these questions:
- Which verified AI crawler identities requested the site?
- How many requests reached the origin versus being blocked at the edge?
- Which URLs returned 200, 3xx, 4xx, or 5xx responses?
- Did requests concentrate on a homepage, sitemap, robots file, feeds, or substantive content?
- Did bot traffic increase after a site migration, WAF change, or robots update?
The distinction is especially important where a crawler appears permitted but no traffic is observed. That can mean the provider has not selected the site for a crawl, the log window is too narrow, the crawler is served at a CDN layer not represented in origin logs, or another technical control is preventing access.
For a worked example, a site might allow OAI-SearchBot in robots.txt, return a 200 on a product page in browser testing, and still show zero verified bot requests because its WAF presents an interstitial challenge to automated requests. The robots policy is correct; the effective delivery path is not.
Staging sites require a different AI-crawler policy
Production and staging should not inherit the same bot permissions by accident. A staging domain can expose unreleased copy, client data, test inventory, or duplicate pages—yet it may also be needed for an agency’s technical review before launch.
A common safe pattern is to protect staging with authentication or network restrictions, then use controlled audit access. Robots rules can add a crawler preference, but they are not a confidentiality control. A Disallow: / line does not make a publicly reachable staging page private.
This is an area where tool choice affects workflow. Crawling a staging website with Screaming Frog or Audra requires attention to authentication, rendering, and crawl scope, not simply a robots setting. The same technical discipline prevents a staging URL from being mistaken for an AI-visibility opportunity.
For client work, reports should state whether a blocked environment was intentionally inaccessible, accidentally restricted, or simply outside the agreed test scope. That preserves context and prevents a correct security control from being presented as a technical SEO defect.
From crawler settings to client-ready recommendations
The 118-site sample suggests that many AEO vendors understand the basic choice to avoid blanket AI-crawler blocks. That should not lead agencies to copy those configurations wholesale. A publisher may have legal, commercial, privacy, licensing, server-capacity, or brand reasons to restrict particular bots.
Instead, an audit recommendation should connect policy to purpose. Here is a concise reporting format:
| Finding | Evidence | Risk or opportunity | Recommended action |
|---|---|---|---|
Search bot blocked from /guides/ | Specific Disallow rule matches tested URL | Key informational pages may be unavailable to that crawler | Confirm intent; remove or narrow rule if visibility is desired |
| Bot allowed but WAF returns 403 | CDN logs and test response | Policy and effective access conflict | Create a verified allow path with security team |
| llms.txt returns 200 but lists old URLs | File inspection and redirects | Curated information is stale | Update links or remove unsupported claims |
| No answer-engine mention for high-value prompt | Prompt test on a defined date | No direct evidence of visibility | Improve source page usefulness and retest over time |
This approach gives the client something actionable and avoids unsupported promises. It also makes comparisons between websites more meaningful. An AEO audit tool for client websites should help preserve evidence, tested URLs, dates, and recommendations—not just generate a generic score.
A sensible policy for the next audit cycle
The right default is neither “allow every AI bot” nor “block every AI bot.” It is a documented, reviewable policy that maps each known crawler to a business decision and checks whether production systems implement that decision.
For many commercial sites, the baseline can be simple:
- allow search-related crawlers where the organization wants discoverability;
- make a separate decision on training-related crawlers;
- protect staging and private environments with real access controls;
- investigate WAF or CDN blocks that contradict approved policy;
- test answer visibility using a stable prompt set and dated screenshots or exports;
- review logs periodically, particularly after platform, security, or robots changes.
The finding that nearly all 118 AEO vendors left AI crawler access open is useful context, not a prescription. The mature practice is to validate what a site says, what bots can fetch, what logs verify, and what answer engines actually surface. That is the difference between checking a file and performing an AI crawler audit.
FAQ
What is an AI crawler audit?
An AI crawler audit reviews whether named AI-related crawlers are permitted and able to access important website content. It usually includes robots.txt rules, page-level directives, HTTP responses, CDN or WAF behavior, and server-log verification. A complete audit also separates crawl access from actual appearance in answer-engine responses.
Does allowing an AI crawler guarantee a ChatGPT or Perplexity citation?
No. Allowing a crawler removes one possible technical barrier, but it does not guarantee crawling, indexing, retrieval, or citation. Answer engines may use different sources and ranking systems for each query. Content quality, topical relevance, authority, page accessibility, and the engine’s own product behavior all affect outcomes.
Does llms.txt improve AI search visibility?
There is no established public evidence that simply publishing llms.txt improves citations in major AI answer engines. The file is a proposed Markdown convention for presenting curated website information. It can be useful operationally, but it should be tested as a supplementary artifact rather than treated as a proven ranking factor.
Should a website block GPTBot but allow OAI-SearchBot?
That depends on the organization’s policy. OpenAI documents separate robots controls for GPTBot and OAI-SearchBot, allowing site owners to make different choices about training-related use and search-related access. The appropriate setup should be approved by the site owner’s legal, content, security, and marketing stakeholders.
How can an agency verify whether an AI crawler really visited a client site?
Review server, CDN, and WAF logs over a defined period, then validate bot identities using the provider’s published guidance where available. Record request volume, requested URLs, response codes, and edge blocks. A user-agent string alone is not sufficient evidence because it can be spoofed by ordinary HTTP clients.