technical seosite auditcrawl speedseo crawlerswebsite performance

Site Audit Crawl Speed: Semrush vs Ahrefs vs Screaming Frog

A practical comparison of crawl-speed controls, safe testing, scope planning, and reporting outputs across leading cloud and desktop site-audit tools.

· 15 min read

A 10,000-URL audit limited to 1 URL every 2 seconds has a theoretical minimum request time of about 5 hours and 33 minutes before redirects, rendering, or server delays are counted. Site audit crawl speed controls therefore determine more than how quickly a dashboard fills: they help teams finish reliable audits within a maintenance window without generating 429 responses, 5XX errors, or misleading timeout findings. For a broader look at the desktop-versus-cloud trade-off, see Screaming Frog vs Ahrefs Site Audit.

This comparison gives SEO consultants, agencies, and site owners a concrete way to select and configure a crawler. It compares Semrush, Ahrefs, Screaming Frog, and Sitebulb on rate controls, scope, rendering, reporting, and operational risk. Audra is included where a local-first, client-ready audit spanning technical checks, performance, accessibility, links, and AI answer-engine visibility is the required output.

ToolCrawl-speed modelRelevant audit outputsPricing treatment as of 10 November 2026Best fit
Semrush Site AuditMinimum, 1 URL per 2 seconds, or respect a crawl-delay settingSite Health score, issue lists, crawl statistics, thematic reportsPlan prices and allowances change; verify on Semrush’s current pricing pageScheduled cloud projects and teams already using Semrush
Ahrefs Site AuditMaximum URLs per minute, with verification-dependent limitsHealth Score, issue reports, link and page reports, crawl historySubscription limits vary by plan and accountCloud auditing alongside Ahrefs data
SitebulbThreads/Chrome instances plus an HTML URLs-per-second capHints, audit scores, URL Explorer, visual crawl mapsSubscription terms vary by region and billing periodTechnical audits requiring explanatory visualisations
Screaming Frog SEO SpiderThreads or URLs requested per second, plus granular limitsInternal, Response Codes, Page Titles, Canonicals, Redirect Chains, exportsFree and paid licence terms should be checked on the vendor pageMigrations, list crawls, custom extraction, forensic work
AudraLocal audit workflow; safe rate depends on the target hostAI visibility, SEO, performance, accessibility, best-practice, and link findingsNo recurring-subscription claim is made here; check product terms before purchaseWhole-site audits and actionable client reporting

Exact price comparisons age quickly and can be affected by currency, billing frequency, seats, crawl credits, and account allowances. Rather than presenting a mixture of exact and generic figures, this page treats current vendor pricing as a purchase-time verification task. The cited vendor pages are the appropriate source for live terms.

Site audit crawl speed is not Google crawl speed

“Crawl speed” is often used for three distinct measures:

  • Audit request rate: requests made by Semrushbot, AhrefsBot, a desktop crawler, or another selected audit user agent.
  • Crawl scope and depth: the URLs included and the number of internal-link hops required to discover them.
  • Google crawl efficiency: Googlebot’s ability to discover and process a site’s URLs.

Only the first is directly controlled in an audit tool. Raising a third-party crawler from 1 to 5 requests per second does not make Googlebot crawl or index a site more quickly.

Google’s crawl budget guidance says crawl-budget management is mainly a concern for very large sites or sites with frequently changing pages. Google Search Console’s Crawl Stats report remains the evidence source for Googlebot request volume, response patterns, and host-status problems. An audit report should therefore say, for example, “the Semrush crawl encountered elevated 5XX responses at the selected rate,” rather than claim that a change to Semrush settings will improve Google indexing.

That distinction matters in client work. If a 10,000-URL crawl is slow because median origin response time is high, the defensible finding is an application or hosting constraint. The useful follow-up is to investigate slow templates, cache misses, database calls, or CDN configuration—not simply to increase crawler concurrency.

Site audit crawl speed: Semrush vs Ahrefs

Semrush and Ahrefs run managed, cloud-based crawlers. Their rate controls are useful, but both vendors advise choosing a speed that the site can handle rather than treating a maximum as a default.

Semrush Site Audit settings and crawl-delay qualification

Semrush documents three crawl-speed choices in its crawl speed documentation: Minimum, 1 URL per 2 seconds, and Respect robots.txt crawl-delay. In Semrush’s Site Audit project configuration, “Minimum” means the Semrush Site Audit crawler uses its fastest available setting rather than applying a Crawl-delay value found in robots.txt.

That statement needs boundaries. It applies to Semrush’s Site Audit crawling behaviour and its configured project speed mode; it is not a general claim about all bots or all robots.txt parsers. Crawl-delay is not part of a universally adopted robots.txt standard, and support can vary by crawler. It also does not override a Disallow rule: a blocked URL remains blocked according to the crawler’s robots handling.

Semrush says it can apply a maximum crawl delay of 30 seconds when the robots-respecting mode is used. Teams should confirm the current behaviour in Semrush’s configuration interface and documentation before a time-sensitive crawl, because settings and bot implementation can change.

A practical Semrush workflow for a 40,000-URL retailer is to begin with XML sitemap URLs or an imported inventory of canonical category and product pages. That initial project can expose missing titles, non-200 pages, broken internal links, and indexability patterns without first spending hours on filter combinations and internal-search URLs. A broader discovery crawl can then investigate architecture and URL generation.

Ahrefs Site Audit settings

Ahrefs’ crawl-speed guidance states that Site Audit begins with a 30-URLs-per-minute maximum and that verified sites can access substantially higher limits, up to 30,000 URLs per minute under the documented conditions. The same guidance frames the right setting as the fastest one that does not affect regular visitors.

That 30,000 figure is a configurable ceiling, not a measured crawl result or a safe recommendation for every host. A WordPress store on shared hosting, for example, may return 429 or 503 responses well below 30 requests per minute if uncached product queries are expensive. Conversely, a static site served from a robust CDN may tolerate much more.

Ahrefs also documents Always-on Audit as a separate continuous-crawling workflow. Its rate and availability depend on the project’s plan, credits, and boost configuration. It should not be compared directly with a one-time maximum-rate crawl: the former is intended to detect changes over time, while the latter builds a faster baseline.

Neither platform’s broad SEO dataset automatically makes its crawl report more complete for every job. The decision should depend on whether the team needs recurring cloud history, project-level reports, and adjacent platform data, or whether it needs desktop-level control for a specific technical investigation.

Sitebulb vs Screaming Frog: desktop rate control

Desktop crawlers give an operator more direct control over threads, rendering, machine resources, and collection rules. That is valuable for migrations and diagnostics, but it also means the operator is responsible for rate testing and local capacity.

Sitebulb defaults, rendering, and timeouts

Sitebulb’s Crawler Settings documentation describes controls for the number of HTML threads or Chrome instances and a maximum HTML URLs-per-second setting. Its related crawl-speed guide explains that the configured cap is a maximum, not a promise that every site will sustain that rate.

The vendor documentation available on 10 November 2026 describes a default of five threads or Chrome instances and a five-HTML-URLs-per-second cap. These are documented product defaults, not universal best-practice figures; users should check their installed version because desktop software settings can change between releases.

Sitebulb’s documentation also describes a 10-second TTFB timeout in the relevant crawler configuration. Editorially, that means a timeout is evidence that the selected crawl could not obtain an initial response in that window. It does not independently prove that every normal visitor has the same experience. Validate repeatable failures with browser tests, server logs, or real-user monitoring.

HTML and Chrome crawling answer different questions. HTML crawling is usually appropriate for a broad baseline of status codes, canonicals, headings, links, and robots directives. Chrome crawling is needed when JavaScript changes meaningful content, links, canonical tags, or structured data. A sensible sample might compare 20 product pages, 10 category pages, and 10 editorial pages in both modes before committing thousands of URLs to rendering.

Screaming Frog controls and machine guidance

Screaming Frog SEO Spider allows speed control through concurrent threads or URLs requested per second. Its configuration guide also covers limits for crawl depth, response size, redirects, query strings, rendering, and other scope decisions.

The vendor’s SEO Spider page publishes current free and licence terms. Those terms should be checked directly rather than copied into a comparison where they may become outdated.

Screaming Frog’s guidance has commonly used 8 GB RAM as a planning point for crawls below 200,000 URLs and 16 GB RAM plus SSD storage as a useful starting point for very large crawls. These are capacity guidelines, not guarantees. A 20,000-URL JavaScript crawl with screenshots, JavaScript rendering, custom extraction, and API integrations can consume more resources than a much larger HTML-only crawl.

For a migration, Screaming Frog’s list mode is often more meaningful than maximum discovery speed. An operator can upload 5,000 legacy URLs, crawl them, export status codes, redirect destinations, canonical targets, and chains, then hand developers a finite remediation list. That is a specific output that supports a migration recommendation, rather than a generic claim of “better reporting.”

A worked crawl-time comparison

No vendor-neutral benchmark can state that one crawler will always finish first. Results vary with DNS, host capacity, response times, rendering, redirects, rate limiting, URL count, and whether the crawler receives complete HTML. A transparent calculation is more useful than an invented benchmark.

Assume a defined 10,000-URL HTML-only inventory with one request per URL, no redirects, and an origin that can respond fast enough to sustain the configured rate:

Configured rateCalculationTheoretical request-time floorWhat can make it longer
1 URL per 2 seconds10,000 × 2 seconds5h 33m 20sTTFB, retries, redirects, queue overhead
5 URLs per second10,000 ÷ 533m 20sThread availability and server response time
30 URLs per minute10,000 ÷ 305h 33m 20sPlatform scheduling and crawl processing
300 URLs per minute10,000 ÷ 30033m 20sHost throttling, 429s, 5XXs, WAF rules

This is a rate calculation, not a vendor performance test. It intentionally excludes rendering and extra requests. If each URL redirects once, the crawler processes roughly 20,000 HTTP responses, doubling the simple request count. If Chrome rendering loads scripts, images, CSS, and API calls, the total network work can rise much further.

A useful real-world test records four values for each tool and setting: URL inventory size, elapsed time, response-code distribution, and server-impact signals. A result such as “5 URLs per second, 9,842 completed URLs, 31 429 responses, 18 minutes” is actionable. “Fast crawl” is not.

Why faster settings can reduce audit quality

A rate control helps only when the crawler is waiting on its own rate limit. It cannot correct a slow origin, an expensive application route, DNS failures, a restrictive WAF, or a URL queue inflated by faceted navigation.

Common bottlenecks include:

  • Server response time: a 3-second initial response limits how rapidly threads can become available.
  • JavaScript rendering: a browser crawler must load and execute resources before assessing rendered output.
  • Rate limiting: 429, 403, CAPTCHA, and 5XX responses can make a report incomplete or misleading.
  • URL traps: calendars, parameter combinations, session IDs, and internal search can multiply crawl scope.
  • Local resources: CPU, RAM, disk throughput, and network bandwidth influence desktop crawlers.

The reporting implication is specific. A client-ready report should distinguish “broken URL confirmed at conservative rate” from “URL timed out during an aggressive crawl.” It should include affected URLs, status code or timeout type, source page where available, and the selected crawler configuration. Those details let developers reproduce the issue.

For teams that need a consolidated local audit rather than separate exports, Audra’s local-first workflow can group technical SEO, performance, accessibility, best-practice, broken-link, and AI answer-engine visibility findings in one review. It is designed for macOS and Windows. It should not be positioned as a substitute for every Screaming Frog configuration, particularly custom extraction and highly specialised migration testing.

Safe crawl-speed testing and scope design

The best rate is established through controlled testing, not chosen from the largest available number. A six-step process keeps the test useful:

  1. Define the question. Is the aim to validate sitemap URLs, discover architecture, test redirects, or assess rendered templates?
  2. Build a representative sample. Start with 100 to 500 URLs across high-traffic templates, not just the homepage.
  3. Choose a conservative rate. Record the exact tool, version where relevant, user agent, rendering mode, and rate setting.
  4. Monitor production evidence. Check CDN dashboards, server logs, application monitoring, database load, and visitor-facing latency during the test.
  5. Increase one setting at a time. Raise either threads, URLs per second, or URLs per minute in measured increments.
  6. Stop on warning signals. Reduce the rate when 429s, 5XXs, timeouts, unusual latency, or incomplete responses rise.

Scope control often produces a greater speed improvement than concurrency. Keep canonical products, services, articles, and location pages in the primary audit. Audit XML sitemap URLs separately. Treat filter combinations and internal-search pages as a review class: they should only be excluded after confirming they do not have intended search value. Crawl legacy URLs as a dedicated list during migrations.

For a detailed migration workflow, the Screaming Frog vs Audra migration audit guide explains why a redirect-list crawl and a broader client-facing audit can be complementary deliverables.

Reporting outputs that justify tool selection

Tool recommendations should be tied to outputs, not broad descriptions. Semrush can be appropriate when a team needs scheduled project reporting, issue categories, crawl statistics, and sitemap or imported-URL project sources. Ahrefs can fit a team that needs Site Audit health and issue reporting alongside its existing Ahrefs workflow and crawl history.

Sitebulb is useful when its Hints, URL Explorer, and visual crawl maps help explain a technical relationship, such as deep pages accessible only through a weak internal-link path. Screaming Frog is suitable when exports from Response Codes, Redirect Chains, Canonicals, Page Titles, or Custom Extraction are needed for developer handoff.

Audra is most relevant when the report needs to bring multiple audit disciplines together: technical SEO findings, performance and accessibility checks, best-practice signals, broken links, and AI answer-engine visibility. A recommendation should still name the affected page group and finding rather than relying on a composite score alone.

For example, “replace 214 internal links that point to redirected URLs” is a testable action. “Improve crawl efficiency” is only a theme until the report identifies the source URLs, destinations, redirect chains, and affected templates.

Which should you choose?

Choose Semrush Site Audit when the team already uses Semrush and needs scheduled cloud projects, issue reports, and controlled crawl-source selection. Use robots-respecting mode where the site publishes a meaningful crawl-delay instruction, but confirm the project mode and bot-specific behaviour before relying on it.

Choose Ahrefs Site Audit when verified ownership, cloud crawl history, and the surrounding Ahrefs workflow matter. Treat high URLs-per-minute limits as a capacity to test, not a default configuration.

Choose Sitebulb when visual explanations, crawl maps, and deliberate HTML-versus-Chrome choices will help a technical consultant communicate the cause and impact of an issue.

Choose Screaming Frog when the work requires list mode, custom extraction, authentication, precise crawl restrictions, or migration validation. It is particularly strong when the deliverable is a CSV or spreadsheet of exact URLs and technical fields for developers.

Choose Audra when a local-first desktop audit and a consolidated, actionable report across AI visibility, SEO, performance, accessibility, best practices, and links is the primary need. Teams evaluating cloud-crawler workflows can also review this Ahrefs Site Audit alternative overview.

Verdict

The best crawl speed is the highest tested rate that returns representative data without degrading the site or distorting the audit. For most projects, defining the URL inventory, testing a conservative sample, monitoring production signals, and increasing gradually produces more reliable results than selecting the maximum available setting.

The tool choice should follow the deliverable. Use cloud platforms for recurring project audits and integrated workflows; use desktop crawlers for specialised investigation and export control; use a consolidated local audit where technical, performance, accessibility, link, and AI-visibility findings need to be reviewed together.

FAQ

How can a site audit crawl finish faster without overloading the server?

Reduce unnecessary scope before increasing concurrency. Start with 100 to 500 representative URLs or a sitemap inventory, then monitor server logs, CDN data, response codes, and visitor latency. Increase one rate setting at a time. If 429s, 5XXs, timeouts, or latency spikes appear, the crawl is not safely faster and should be reduced.

What is the maximum Ahrefs Site Audit crawl speed?

Ahrefs documents a default maximum of 30 URLs per minute and, for verified sites under its documented conditions, a ceiling of up to 30,000 URLs per minute. That ceiling is not a benchmark result or universal recommendation. A safe rate depends on server capacity, response time, rate limits, URL complexity, and whether normal visitors are affected.

Does Semrush Minimum crawl speed ignore robots.txt crawl-delay?

For Semrush Site Audit’s documented project speed options, Minimum uses the fastest available crawl setting rather than applying robots.txt Crawl-delay. This is specific to Semrush’s Site Audit crawler and configuration, not a rule for every bot. Because crawl-delay support is not universal, teams should verify current Semrush documentation and test conservatively on sensitive infrastructure.

Does increasing audit crawl speed improve Google crawl budget?

No. A faster Semrush, Ahrefs, Sitebulb, or Screaming Frog crawl changes only that tool’s request rate. Googlebot is governed by Google’s systems and the site’s availability, performance, URL inventory, and discoverability. Resolving server errors, redirect chains, wasteful URL generation, and weak internal linking can support crawl efficiency, but audit-tool speed settings do not control Googlebot.

Should a JavaScript website always use Chrome rendering?

Not necessarily. Chrome rendering is appropriate when important content, links, canonical tags, or structured data appear only after JavaScript executes. It also takes longer and places different demands on the target site and local machine. Compare HTML and rendered output on representative templates first, then apply rendering only where it changes the audit outcome.

Sources