429 Errors During Crawls: A Diagnostic Playbook for SEO Audits
A practical process for separating audit-crawler throttling from genuine Googlebot availability problems, fixing the responsible layer, and proving that crawling has recovered.
· 15 min read
A crawl that returns hundreds of 429 responses can look like a major technical SEO incident, even when normal visitors and Googlebot still receive 200 responses. This guide explains how to diagnose 429 errors during crawls, identify the layer imposing the limit, apply the least risky fix, and prove recovery through a controlled re-crawl, logs, and Google Search Console.
The question surfaced in an SEO community discussion after an auditor saw a sharp rise in 429s across both Screaming Frog and Semrush despite using rate limits. That pattern can indicate stricter server-side controls, but it does not by itself prove that a website is broken, that Cloudflare is responsible, or that Google is blocked.
What a 429 response means during a crawl
HTTP 429 Too Many Requests is the status code a server uses when it wants a client to slow down. RFC 6585 defines it as a rate-limiting response and allows the server to include a Retry-After header telling the client how long to wait before trying again.
For an SEO audit, the key word is client. A 429 delivered to a desktop crawler running from one office IP may be a completely separate event from the response delivered to Googlebot, a logged-in shopper, or a CDN cache.
A crawler can trigger limits surprisingly quickly. For example, a site audit set to 10 crawl threads may request HTML pages, redirects, CSS, JavaScript, images, sitemaps, canonical URLs, and JavaScript-rendering resources in overlapping bursts. A site with client-side rendering can turn one apparent page request into several requests to API endpoints. Crawlers differ in how much they let you throttle; the best SEO crawlers roundup notes the speed controls to check before you choose one.
The first working assumption should therefore be:
- A 429 for an audit crawler is evidence of a rate limit, not proof of an indexing failure.
- A repeated 429 for verified Googlebot is a Google availability concern.
- A 429 on
/api/, search, account, cart, or filter URLs may point to a targeted rule rather than a sitewide capacity problem. - A 429 that coincides with slow response times or 5xx errors may indicate an origin resource issue rather than a security rule.
Start with a 15-minute triage before changing crawl speed
Lowering crawl speed is a sensible containment step, but it should not be the entire investigation. It can hide the underlying issue and make a crawl take many hours longer than necessary.
Begin with the 429 sample from the audit tool. Export the affected URLs, response headers, timestamps, response times, and user agent where available. Then compare those records with server access logs, CDN logs, WAF events, and application monitoring for the same five- to 15-minute window.
Check who received the 429
Separate requests into at least four groups:
- The audit crawler — for example, Screaming Frog, Semrush Site Audit, or an Audra local audit.
- Verified Googlebot — not a request merely claiming to use a Googlebot user agent.
- Normal visitor traffic — browser sessions, uptime checks, or synthetic monitoring.
- Other automated traffic — AI bots, scrapers, monitoring vendors, feeds, integrations, or internal jobs.
Google provides verification methods for crawler requests, including reverse DNS validation and published IP-range checks. A user-agent string alone is not enough to allowlist a purported Googlebot.
If only the SEO tool receives 429s and normal browser requests remain fast and successful, treat this initially as an audit-tool throttling problem. If verified Googlebot and normal visitors also see 429s, move directly to host capacity, CDN/WAF, and application diagnosis.
Compare request rate and timing
A useful log query groups requests by minute, status code, user agent, IP, host, and URL path. The goal is to answer a concrete question: did 429s begin after a burst from one client, or do they occur at a stable low rate?
For example, 80 requests in one minute from a single IP followed by 429s strongly suggests a per-IP threshold. A pattern of 429s at two requests per second on expensive faceted URLs suggests a path-specific or application-level limit. A mix of slow 200s, 429s, and 503s suggests that the origin is reaching its serving capacity.
Identify the layer issuing the rate limit
The safest fix depends on where the response is generated. A 429 is an HTTP outcome, not a diagnosis.
Origin server or hosting stack
A web server, PHP worker pool, database connection limit, managed host, or load balancer may return 429 when requests exceed available capacity. Check response headers, error logs, CPU, memory, database wait time, queue length, and upstream response timing.
If a new crawl reveals that the server is already slow before 429s begin, simply raising a WAF threshold can expose the origin to more pressure. In that scenario, reduce crawler concurrency first, then investigate caching, inefficient queries, worker limits, and infrastructure capacity.
CDN or WAF
Cloudflare and similar edge services can apply rate limits based on IP address, hostname, path, request method, headers, or other characteristics. Cloudflare documentation also notes that a rule can be configured to count cached assets, which matters because a crawler may request many static files as well as HTML.
Look for response headers such as server, cf-ray, x-cache, provider-specific request IDs, or a branded error page. Then inspect security events and rate-limit rules using the exact timestamp and source IP. A rule protecting /login may be appropriate; that same rule accidentally matching all URLs below / is not.
Application and API endpoints
JavaScript-heavy sites often create the most confusing crawl results. The initial HTML may return 200 while a product, search, availability, or review API returns 429. A rendered crawl can therefore report incomplete content even though a basic HTML crawl looks clean.
Audit the failed paths separately. Do not solve a product API throttle by increasing crawl threads, and do not treat failed third-party scripts as a primary HTML indexing issue without checking whether page content depends on them.
Platform-specific controls, including Shopify
Shopify documents API rate limits using methods that vary by API, plan, and query cost. That is relevant when a crawl or rendering process reaches custom storefront integrations, app proxies, or administrative endpoints, but it does not mean that every public Shopify page returning 429 is a standard Shopify platform limit.
For a Shopify store, isolate the hostname and path. A 429 on a storefront page, a search app endpoint, and /admin/api/ are three different cases. The evidence may instead point to a third-party app, a CDN rule, or a bot-protection service in front of the store.
How to handle audit-only 429 errors during crawls
When only Screaming Frog, Semrush, Audra, or another audit client is rate-limited, the goal is complete enough coverage without placing unnecessary load on the site.
Run a controlled test rather than repeatedly launching the same full crawl. Start with a representative sample: the homepage, primary navigation, XML sitemap URLs, priority templates, and a small selection of deep product or article pages.
Use this sequence:
- Set a low concurrency level, such as one or two crawl threads, with a delay between requests if the crawler supports it.
- Disable JavaScript rendering temporarily to determine whether rendering-related assets or APIs trigger the limit.
- Restrict the crawl to HTML when the immediate purpose is a technical indexability check.
- Re-crawl the original 429 URLs after the delay specified in
Retry-After, if present. - Increase concurrency one small step at a time only after the 429 rate remains near zero.
This approach finds an operating ceiling instead of guessing. If two threads work but five threads immediately trigger 429s, record that evidence in the audit. It gives a developer or host a reproducible scenario: specific user agent, IP, path set, thread count, request rate, and timestamps.
For local-first audits, this is especially useful because the operator can control the machine, IP, crawl settings, and test scope. Audra can be used to establish a baseline across technical SEO, performance, accessibility, and links, then re-run a constrained audit after the limiting layer is changed.
A crawler should not compensate for rate limiting by silently omitting critical URLs. Preserve the 429 URL list, prioritize canonical indexable pages, and schedule a low-rate retry queue. That avoids mistaking missing data for healthy pages.
When 429s can affect Googlebot and crawl budget
Google treats significant numbers of 429, 500, and 503 responses as a signal to reduce crawling across the affected hostname. Google’s current guidance says it automatically increases crawl activity again as those errors decrease.
That makes persistent 429s different from an isolated audit-tool block. If Googlebot receives them repeatedly, fewer new pages may be discovered and existing pages may be refreshed less often. Google also warns that returning 429 or 503 responses for more than roughly two days can lead to affected URLs being dropped from the index.
This is a host-availability problem first and a crawl-budget problem second. It deserves priority when the site is large, frequently updated, ecommerce-heavy, or dependent on fast inventory and price refreshes.
Google’s crawl budget guidance is most relevant to large or frequently changing sites. Smaller sites should still fix persistent 429s, but they should avoid attributing every indexing delay to “crawl budget” without evidence.
Use Search Console Crawl Stats and logs to separate Googlebot from audit traffic
Google Search Console’s Crawl Stats report is the main confirmation point for Google’s view of host availability. Review it alongside logs; neither source is a substitute for the other.
In Crawl Stats, inspect the host-status trend, crawl requests, average response time, and response breakdown around the dates when 429s rose. A decline in crawl requests combined with rising host-availability issues or response-time problems supports the theory that Googlebot is encountering availability constraints.
Server logs answer the more precise question: which URLs, responses, and user agents were involved? Filter logs for 429 status codes and compare:
- verified Googlebot versus desktop crawler user agents;
- the affected hostname, including
wwwand non-wwwvariants; - HTML document requests versus assets and API calls;
- request volume per minute and median response time;
- the rule, upstream service, or application message responsible for the response.
Use URL Inspection for a small sample of strategic pages after remediation, but do not confuse a successful single inspection with proof that the entire host has recovered. Crawl Stats and logs provide the trend-level evidence.
For broader diagnosis of unstable organic metrics, the method in Google Rank Tracking Broken? How to Verify a Real Ranking Drop is relevant: verify the underlying data source before treating a dashboard change as a ranking or crawling event.
What the Retry-After header does—and does not do
A Retry-After header communicates a wait period after a temporary response such as 429 or 503. It can be expressed as a number of seconds or as an HTTP date.
For an audit crawler, the practical use is straightforward: honor the stated delay before retrying, rather than sending another burst immediately. If the header says Retry-After: 120, a retry after two minutes is safer than an instant retry loop.
For Googlebot, Google documents that 429 and 503 responses can include Retry-After to specify how long the crawler should wait. The header is helpful, but it is not a substitute for restoring service. A site should not rely on repeated 429 responses as a routine crawl-management method.
Two limitations matter:
- Many implementations return 429 without a
Retry-Afterheader, so the client must use sensible exponential backoff and conservative concurrency. - The header does not explain which system imposed the limit. Logs and response metadata still determine whether the source was the CDN, WAF, origin, application, or an upstream API.
Fix the least risky layer instead of blanket-allowlisting bots
The right remediation preserves protections for sensitive endpoints while allowing legitimate crawling of public content. A blanket “allow all bots” rule is rarely the first choice, because user-agent strings are easily spoofed and high request volumes can still overload the origin.
Choose the fix based on the evidence:
| Finding | Least-risky first fix |
|---|---|
| Only one audit IP hits a global per-IP threshold | Reduce threads and add a delay; use a documented audit window if needed. |
| Public HTML routes are caught by a WAF rule | Narrow the rule by path, method, or request characteristic; retain protection on login, checkout, and APIs. |
| API calls fail only during JavaScript rendering | Limit rendering scope, cache where possible, and tune the API-specific rate limit or client behavior. |
| Verified Googlebot receives 429s with slow origin responses | Improve capacity, caching, queries, or worker allocation before relaxing edge protections. |
| A suspected Googlebot is being blocked | Verify it using Google’s process before considering a narrowly scoped exception. |
A content-heavy site may also need crawl-efficiency work. Infinite filters, calendar pages, internal search parameters, and duplicate URL variants can create unnecessary demand. Google identifies faceted navigation and calendar-style URLs among common reasons for sharp crawling increases.
For performance-driven remediation, AI Agents Fix Lighthouse Errors with Chrome DevTools provides a related workflow for diagnosing page-level technical issues before reporting them to a client.
JavaScript, assets, and third-party services need a separate test
A rendered audit may produce more 429s than a non-rendered crawl because the browser requests assets, hydration data, analytics endpoints, product APIs, and third-party widgets. That does not automatically mean Google cannot index the initial HTML.
Run two comparable tests on the same URL sample:
- HTML-only crawl: assesses server responses, canonicals, metadata, indexability, redirects, and links.
- JavaScript-rendered crawl: assesses whether essential rendered content or resource loading introduces failures.
If HTML returns 200 but essential product details disappear because an API returns 429, the issue is still meaningful for users and for rendering. If a nonessential analytics script returns 429 while the HTML and critical content work, document it separately rather than inflating the technical SEO error count.
Google recommends using Crawl Stats to monitor both Googlebot and Web Rendering Service activity when investigating JavaScript search problems. The important distinction is whether the failed request prevents meaningful page content from being served or rendered.
Verification checklist: prove that the fix worked
A rate-limit change is not complete when someone says the rule was adjusted. It is complete when evidence shows that public pages are consistently available at a safe request rate.
Use this post-fix checklist:
- Re-run the original crawl sample with the same user agent, source IP, scope, JavaScript setting, and concurrency that previously produced 429s.
- Confirm sampled priority URLs now return 200 responses and retain their expected canonical, robots, and content behavior.
- Review server, CDN, WAF, and application logs for 429s during the controlled re-crawl.
- Increase threads gradually only if error rates remain stable; do not jump from two threads to 20.
- Check Search Console Crawl Stats over the following days for host availability, response-time, and crawl-request trends.
- Verify a sample of important URLs with URL Inspection if Googlebot was implicated.
- Set an alert for recurring 429s by hostname, path group, user agent class, and response source.
The recovery window varies by site and the nature of the issue. Google states that it raises crawl activity again after 429, 500, and 503 responses decline, but no fixed recovery duration applies to every hostname. Record before-and-after timestamps so the team can distinguish a genuine recovery from normal daily variation.
For client reporting, the best summary is not “429s fixed.” It is: which crawler was affected, which layer limited it, the evidence behind the change, the safe crawl configuration, and whether Googlebot availability was confirmed. That turns a vague crawl error into an auditable operational finding.
FAQ
What causes 429 Too Many Requests errors during a website crawl?
429 errors usually occur when a crawler exceeds a rate limit set by the origin server, CDN, WAF, application, API, or hosting platform. The trigger can be thread count, requests per second, costly JavaScript rendering, repeated requests to filtered URLs, or a path-specific rule. Logs and response headers are needed to identify the responsible layer.
Do repeated 429 errors hurt Google crawling, indexing, or crawl budget?
They can if verified Googlebot receives them repeatedly. Google reduces crawl activity across a hostname after significant 429, 500, or 503 responses, and warns that 429 or 503 responses lasting more than about two days can cause affected URLs to drop from the index. Audit-tool-only 429s do not prove that Google is affected.
How do I reduce 429 errors in an SEO crawler without missing important URLs?
Use a controlled crawl: reduce crawl threads, introduce a delay, start with HTML-only crawling, and prioritize sitemap URLs and indexable canonical pages. Preserve the list of failed URLs and retry it after the indicated Retry-After period. Increase concurrency gradually only after the sample crawl remains stable.
Should I lower crawl speed, increase server resources, or change CDN/WAF rate-limit rules?
Lower crawl speed is the safest immediate action, but it is not always the permanent fix. Change the layer shown by evidence: tune a path-specific WAF rule when it catches public content, increase resources when verified Googlebot hits capacity-related 429s, and adjust crawler behavior when only one audit client is being throttled.
How can I tell whether 429s affect only my audit crawler or Googlebot too?
Compare audit exports with server access logs, CDN/WAF events, and Google Search Console Crawl Stats. Verify suspected Googlebot requests through Google’s documented verification process rather than trusting the user agent. If the 429s are limited to the auditor’s IP and Search Console host availability is stable, the problem is likely audit-specific.
Sources
- https://www.reddit.com/r/SEO/comments/1xcn399/increase_in_429_errors_during_crawls/
- https://developers.google.com/crawling/docs/crawlers-fetchers/reduce-crawl-rate
- https://developers.google.com/search/docs/crawling-indexing/troubleshoot-crawling-errors
- https://developers.google.com/search/docs/monitor-debug/search-console-crawl-stats-report
- https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests
- https://www.rfc-editor.org/rfc/rfc6585
- https://developers.cloudflare.com/waf/rate-limiting-rules/parameters/
- https://shopify.dev/docs/api/usage/limits