Best robots.txt Testers in 2026: 8 Ways to Check Google and AI Bots
Search Console now has a robots.txt report rather than a URL-by-URL tester, so testing a rule means using another tool. Eight robots.txt testers compared by parser, AI-bot coverage, draft testing and site-wide checks.
· 9 min read
The best robots.txt tester for checking a URL against Google's rules is one built on Google's open-source robots.txt parser, such as Tame the Bots, which tests a list of URLs against your live file or a draft and shows the matching rule. TechnicalSEO.com's tester adds a live editor and checks whether a page's CSS, JavaScript and images are blocked. For AI crawlers, Keywords Everywhere and Mencoro test URLs as GPTBot, ClaudeBot and other AI bots. Search Console's robots.txt report tells you which file Google actually fetched and any parsing errors, but it does not test individual URLs. To check bot access across every page of a site, a site auditor such as Audra scores it per page.
Google's documentation adds two details any tester should respect: Google enforces a 500 KiB size limit (content after it is ignored) and generally caches robots.txt for up to 24 hours. Details below were checked on each tool's own page on November 11, 2026.
robots.txt testers at a glance
| Tool | Parser | Live or draft file | AI crawlers | Tests many URLs | Price |
|---|---|---|---|---|---|
| Search Console robots.txt report | Google's own | Live (as Google fetched it) | No | No (file-level report) | Free |
| Google robotstxt library | Google's production parser, open source | Local file | Any user-agent you pass | One URL per run | Free, open source (C++) |
| Tame the Bots | Google's robotstxt library | Live or custom | Choose the user-agent | Yes, a list, across sites | Free; also offers an MCP server |
| TechnicalSEO.com | Not stated | Live or editor | Choose the user-agent | One URL plus its resources | Free |
| Keywords Everywhere | RFC 9309 parsing, in your browser | Live or pasted draft | Yes: Googlebot, GPTBot, ClaudeBot and more | Yes | Free tool |
| Mencoro | Not stated | Live or edited | Yes: Googlebot, Bingbot and AI crawlers | One URL at a time | Free |
| Screaming Frog (custom robots.txt) | Its own | Custom file during a crawl | Any user-agent you configure | Whole crawl | Paid licence ($279/year) |
| Audra | Its own robots.txt check | Live, during an audit | Retrieval and training bots, scored separately | Every crawled page | $19 once (launch price) |
The 8 best robots.txt testers
1. Search Console robots.txt report: best for what Google actually fetched
Search Console's robots.txt report shows which robots.txt files Google found for the top 20 hosts on your site, when each was last crawled, and any warnings or errors; Google ignores lines it cannot parse and uses the rest. You can also request a recrawl of the file in an emergency. It is available only for domain-level properties (a Domain property, or a URL-prefix property without a path).
Use it for: confirming Google sees the file you think it sees. It does not: tell you whether a specific URL is blocked; use URL Inspection or a tester for that.
2. Google's robotstxt library: best for exact Googlebot behaviour
Google open-sourced the robots.txt parser and matcher that Googlebot uses in production (slightly modified), as a C++ library. It includes a small binary for testing one URL and user-agent against a local robots.txt file.
Use it for: settling a disputed rule. It needs: a C++14 compiler and a terminal.
3. Tame the Bots: best for testing many URLs with Google's parser
Tame the Bots' robots.txt Testing Tool uses Google's robots.txt parser and matcher library to check a list of URLs against your live file, or a custom one you paste, and shows whether each is allowed or blocked and by which rule. You can test URLs from several sites at once, and it notes cases such as Googlebot-Image falling back to Googlebot rules. It also offers an MCP server.
Use it for: testing a draft before you publish it.
4. TechnicalSEO.com robots.txt Validator: best for blocked page resources
TechnicalSEO.com's tool checks whether a URL is blocked, which statement blocks it and for which user agent, and whether the page's resources (CSS, JavaScript, images) are disallowed, with both a live mode and an editor.
Use it for: catching blocked CSS or JavaScript that stops Google rendering a page.
5. Keywords Everywhere robots.txt Tester: best quick AI-bot check
Keywords Everywhere's free tester fetches a site's live robots.txt once, then parses it in your browser per RFC 9309 and matches each URL you enter the way crawlers do, showing the exact rule that decided it. Its crawler list includes AI training and browsing bots such as GPTBot and ClaudeBot, and you can paste a draft file to test before deploying.
Use it for: "did we actually block GPTBot?" in seconds.
6. Mencoro robots.txt Tester: best for Google, Bing and AI bots together
Mencoro's free tester checks any URL against your live robots.txt or an edited version for Googlebot, Bingbot and AI crawlers, and shows the rule that allows or blocks it.
Use it for: one URL across several crawlers at once.
7. Screaming Frog custom robots.txt: best for testing a draft across a crawl
Screaming Frog lists custom robots.txt as a paid-licence feature, so you can crawl a site as if a draft file were live and see every URL it would block before you deploy it. The licence is $279 per user per year.
Use it for: migrations and large rule changes, where one-URL tests are not enough.
8. Audra: best for AI-bot access across a whole site
Audra checks robots.txt on every audit and treats the two kinds of AI crawler differently. Access for the retrieval bots that produce citations (ChatGPT-User, OAI-SearchBot and PerplexityBot) is 30% of its AEO score, because blocking them makes every other answer-engine signal moot. Rules for training bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and Bytespider) are reported but never scored as a failure, since blocking them is a legitimate policy choice. Audra also respects robots.txt while it crawls. It is a desktop app for macOS on Apple Silicon and Windows 10/11, at $19 once (launch price; regular price $49, also one-time) after a free 7-day trial.
Use it for: confirming AI-bot access as part of a full SEO and AEO audit.
Which AI crawlers should your robots.txt allow?
It depends on whether you want to be cited, trained on, or both. The vendors document the split:
| Crawler | Vendor's stated purpose | If you block it |
|---|---|---|
| OAI-SearchBot | Surfacing websites in ChatGPT's search features | OpenAI says the site will not be shown in ChatGPT search answers, though it can still appear as a navigational link |
| GPTBot | Crawling content that may be used to train OpenAI's foundation models | Signals the content should not be used for training |
| ChatGPT-User | Certain user-initiated visits in ChatGPT and Custom GPTs; not used for automatic crawling | OpenAI says robots.txt rules may not apply to these user-initiated visits, and ChatGPT-User does not decide whether content appears in Search |
| PerplexityBot | Surfacing and linking websites in Perplexity's search results; Perplexity says it is not used for foundation-model training | Reduces the chance of appearing in Perplexity results |
| Perplexity-User | Fetches made when a user asks Perplexity a question | Perplexity says this fetcher generally ignores robots.txt, because the user requested it |
OpenAI notes that a robots.txt change can take about 24 hours to affect search results. For more on the OpenAI bots, see OpenAI crawlers and robots.txt, and for checking whether a bot that claims to be one really is, verifying AI bot traffic.
FAQ
Does Google Search Console still have a robots.txt tester?
Search Console now offers a robots.txt report instead: it shows which robots.txt files Google found for your top 20 hosts, when they were last crawled and any errors, and lets you request a recrawl in an emergency. To test whether a specific URL is blocked, use URL Inspection or a tester built on Google's parser, such as Tame the Bots.
How do I test if a URL is blocked by robots.txt?
Paste the URL into a tester such as Tame the Bots, TechnicalSEO.com, Keywords Everywhere or Mencoro, choose the user-agent (for example Googlebot or GPTBot), and read which rule allows or blocks it. Test against a draft file before publishing changes.
How do I check whether my robots.txt blocks AI crawlers?
Test a few important URLs as OAI-SearchBot, PerplexityBot, GPTBot and ClaudeBot in a tester with an AI-bot list, such as Keywords Everywhere or Mencoro. To check every page of a site at once, an audit tool such as Audra reports retrieval-bot and training-bot access separately.
How big can a robots.txt file be?
Google enforces a 500 KiB limit and ignores anything after it. Consolidate rules if your file is close to that size.
How long does a robots.txt change take to apply?
Google generally caches robots.txt for up to 24 hours, longer if it cannot refresh the file. OpenAI says about 24 hours for its search results. Search Console's robots.txt report lets you request a recrawl in an emergency.
Sources
- https://support.google.com/webmasters/answer/6062598?hl=en
- https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt
- https://github.com/google/robotstxt
- https://technicalseo.com/tools/robots-txt/
- https://tamethebots.com/tools/robotstxt-checker
- https://keywordseverywhere.com/tools/robots-txt-tester/
- https://mencoro.com/tools/robots-txt-tester/
- https://www.screamingfrog.co.uk/seo-spider/pricing/
- https://developers.openai.com/api/docs/bots
- https://docs.perplexity.ai/guides/bots
- https://audra.greta.sh/