robots.txt testerrobots.txttechnical seoai crawlersgoogle search console

Best robots.txt Testers in 2026: 8 Ways to Check Google and AI Bots

Search Console now has a robots.txt report rather than a URL-by-URL tester, so testing a rule means using another tool. Eight robots.txt testers compared by parser, AI-bot coverage, draft testing and site-wide checks.

· 9 min read

The best robots.txt tester for checking a URL against Google's rules is one built on Google's open-source robots.txt parser, such as Tame the Bots, which tests a list of URLs against your live file or a draft and shows the matching rule. TechnicalSEO.com's tester adds a live editor and checks whether a page's CSS, JavaScript and images are blocked. For AI crawlers, Keywords Everywhere and Mencoro test URLs as GPTBot, ClaudeBot and other AI bots. Search Console's robots.txt report tells you which file Google actually fetched and any parsing errors, but it does not test individual URLs. To check bot access across every page of a site, a site auditor such as Audra scores it per page.

Google's documentation adds two details any tester should respect: Google enforces a 500 KiB size limit (content after it is ignored) and generally caches robots.txt for up to 24 hours. Details below were checked on each tool's own page on November 11, 2026.

robots.txt testers at a glance

ToolParserLive or draft fileAI crawlersTests many URLsPrice
Search Console robots.txt reportGoogle's ownLive (as Google fetched it)NoNo (file-level report)Free
Google robotstxt libraryGoogle's production parser, open sourceLocal fileAny user-agent you passOne URL per runFree, open source (C++)
Tame the BotsGoogle's robotstxt libraryLive or customChoose the user-agentYes, a list, across sitesFree; also offers an MCP server
TechnicalSEO.comNot statedLive or editorChoose the user-agentOne URL plus its resourcesFree
Keywords EverywhereRFC 9309 parsing, in your browserLive or pasted draftYes: Googlebot, GPTBot, ClaudeBot and moreYesFree tool
MencoroNot statedLive or editedYes: Googlebot, Bingbot and AI crawlersOne URL at a timeFree
Screaming Frog (custom robots.txt)Its ownCustom file during a crawlAny user-agent you configureWhole crawlPaid licence ($279/year)
AudraIts own robots.txt checkLive, during an auditRetrieval and training bots, scored separatelyEvery crawled page$19 once (launch price)

The 8 best robots.txt testers

1. Search Console robots.txt report: best for what Google actually fetched

Search Console's robots.txt report shows which robots.txt files Google found for the top 20 hosts on your site, when each was last crawled, and any warnings or errors; Google ignores lines it cannot parse and uses the rest. You can also request a recrawl of the file in an emergency. It is available only for domain-level properties (a Domain property, or a URL-prefix property without a path).

Use it for: confirming Google sees the file you think it sees. It does not: tell you whether a specific URL is blocked; use URL Inspection or a tester for that.

2. Google's robotstxt library: best for exact Googlebot behaviour

Google open-sourced the robots.txt parser and matcher that Googlebot uses in production (slightly modified), as a C++ library. It includes a small binary for testing one URL and user-agent against a local robots.txt file.

Use it for: settling a disputed rule. It needs: a C++14 compiler and a terminal.

3. Tame the Bots: best for testing many URLs with Google's parser

Tame the Bots' robots.txt Testing Tool uses Google's robots.txt parser and matcher library to check a list of URLs against your live file, or a custom one you paste, and shows whether each is allowed or blocked and by which rule. You can test URLs from several sites at once, and it notes cases such as Googlebot-Image falling back to Googlebot rules. It also offers an MCP server.

Use it for: testing a draft before you publish it.

4. TechnicalSEO.com robots.txt Validator: best for blocked page resources

TechnicalSEO.com's tool checks whether a URL is blocked, which statement blocks it and for which user agent, and whether the page's resources (CSS, JavaScript, images) are disallowed, with both a live mode and an editor.

Use it for: catching blocked CSS or JavaScript that stops Google rendering a page.

5. Keywords Everywhere robots.txt Tester: best quick AI-bot check

Keywords Everywhere's free tester fetches a site's live robots.txt once, then parses it in your browser per RFC 9309 and matches each URL you enter the way crawlers do, showing the exact rule that decided it. Its crawler list includes AI training and browsing bots such as GPTBot and ClaudeBot, and you can paste a draft file to test before deploying.

Use it for: "did we actually block GPTBot?" in seconds.

6. Mencoro robots.txt Tester: best for Google, Bing and AI bots together

Mencoro's free tester checks any URL against your live robots.txt or an edited version for Googlebot, Bingbot and AI crawlers, and shows the rule that allows or blocks it.

Use it for: one URL across several crawlers at once.

7. Screaming Frog custom robots.txt: best for testing a draft across a crawl

Screaming Frog lists custom robots.txt as a paid-licence feature, so you can crawl a site as if a draft file were live and see every URL it would block before you deploy it. The licence is $279 per user per year.

Use it for: migrations and large rule changes, where one-URL tests are not enough.

8. Audra: best for AI-bot access across a whole site

Audra checks robots.txt on every audit and treats the two kinds of AI crawler differently. Access for the retrieval bots that produce citations (ChatGPT-User, OAI-SearchBot and PerplexityBot) is 30% of its AEO score, because blocking them makes every other answer-engine signal moot. Rules for training bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and Bytespider) are reported but never scored as a failure, since blocking them is a legitimate policy choice. Audra also respects robots.txt while it crawls. It is a desktop app for macOS on Apple Silicon and Windows 10/11, at $19 once (launch price; regular price $49, also one-time) after a free 7-day trial.

Use it for: confirming AI-bot access as part of a full SEO and AEO audit.

Which AI crawlers should your robots.txt allow?

It depends on whether you want to be cited, trained on, or both. The vendors document the split:

CrawlerVendor's stated purposeIf you block it
OAI-SearchBotSurfacing websites in ChatGPT's search featuresOpenAI says the site will not be shown in ChatGPT search answers, though it can still appear as a navigational link
GPTBotCrawling content that may be used to train OpenAI's foundation modelsSignals the content should not be used for training
ChatGPT-UserCertain user-initiated visits in ChatGPT and Custom GPTs; not used for automatic crawlingOpenAI says robots.txt rules may not apply to these user-initiated visits, and ChatGPT-User does not decide whether content appears in Search
PerplexityBotSurfacing and linking websites in Perplexity's search results; Perplexity says it is not used for foundation-model trainingReduces the chance of appearing in Perplexity results
Perplexity-UserFetches made when a user asks Perplexity a questionPerplexity says this fetcher generally ignores robots.txt, because the user requested it

OpenAI notes that a robots.txt change can take about 24 hours to affect search results. For more on the OpenAI bots, see OpenAI crawlers and robots.txt, and for checking whether a bot that claims to be one really is, verifying AI bot traffic.

FAQ

Does Google Search Console still have a robots.txt tester?

Search Console now offers a robots.txt report instead: it shows which robots.txt files Google found for your top 20 hosts, when they were last crawled and any errors, and lets you request a recrawl in an emergency. To test whether a specific URL is blocked, use URL Inspection or a tester built on Google's parser, such as Tame the Bots.

How do I test if a URL is blocked by robots.txt?

Paste the URL into a tester such as Tame the Bots, TechnicalSEO.com, Keywords Everywhere or Mencoro, choose the user-agent (for example Googlebot or GPTBot), and read which rule allows or blocks it. Test against a draft file before publishing changes.

How do I check whether my robots.txt blocks AI crawlers?

Test a few important URLs as OAI-SearchBot, PerplexityBot, GPTBot and ClaudeBot in a tester with an AI-bot list, such as Keywords Everywhere or Mencoro. To check every page of a site at once, an audit tool such as Audra reports retrieval-bot and training-bot access separately.

How big can a robots.txt file be?

Google enforces a 500 KiB limit and ignores anything after it. Consolidate rules if your file is close to that size.

How long does a robots.txt change take to apply?

Google generally caches robots.txt for up to 24 hours, longer if it cannot refresh the file. OpenAI says about 24 hours for its search results. Search Console's robots.txt report lets you request a recrawl in an emergency.

Sources