How to Crawl Large Websites: Audra vs Screaming Frog, Sitebulb and Distributed Crawlers

Screaming Frog’s published large-site guidance estimates approximately 2 million URLs in database-storage mode with 4 GB of memory allocated; its approximately 10 million-URL example at 16 GB also assumes a 500 GB SSD. Those figures illustrate why how to crawl large websites is primarily a scoping and storage decision, not a contest to fetch the most URLs.

The practical payoff for SEO consultants, agencies, marketers, and site owners is a defensible audit: one that finds technical SEO, performance, accessibility, link, and AI-answer visibility issues without exhausting local resources, triggering a parameter trap, or handing a client an unusable export.

Define “large” before choosing a crawler

A large website is an operational category rather than one universal URL count. A 50,000-URL ecommerce site with JavaScript rendering, faceted navigation, image resources, and unlimited filter combinations can be harder to audit than a 500,000-URL static publishing site.

A useful planning model is:

• Up to tens of thousands of URLs: a whole-site local audit may be realistic when crawl speed and rendering are controlled.
• Around 100,000 to 500,000 URLs: exclusions, storage mode, and a clear audit objective become essential.
• From 500,000 into the millions: database-backed crawling, segmented audits, or purpose-built cloud/infrastructure options deserve consideration.
• Several million URLs and above: first establish whether the work is an SEO audit or a data-collection system. Those are different jobs.

Screaming Frog documents memory storage as more suitable for smaller sites and database storage as the mode for much larger websites. Its published estimates are approximate because page size, link counts, JavaScript rendering, resource collection, server response time, disk speed, and computer configuration all affect capacity. Specifically, its guide estimates about 2 million URLs with 4 GB of allocated RAM in database storage; the 16 GB example of roughly 10 million URLs carries the stated requirement for a 500 GB SSD.

Before selecting software, define the question the crawl must answer. A technical SEO crawl for large websites may need to identify duplicate indexable category URLs, broken internal links, conflicting canonicals, weak security headers, or failing core templates. It does not always need every discoverable URL.

How to crawl large websites without wasting capacity

The most reliable large website crawling practice is to reduce the crawlable universe before starting. That is not an incomplete audit when the exclusions reflect URLs that users and search engines should not need to reach.

Start with a documented scope:

1. Choose intentional seeds. Use XML sitemaps, category roots, content hubs, a verified URL export, or a specific directory rather than relying only on homepage discovery.
2. Exclude known low-value paths. Common examples include internal search results, carts, customer-account paths, session IDs, tracking parameters, print pages, and test URLs.
3. Create parameter rules. An ecommerce crawl might retain /mens-shoes/ while excluding ?sort=, ?sessionid=, ?utm_, and non-indexable filter combinations.
4. Select only necessary crawl data. Sitebulb’s large-site guidance recommends a sample audit and selecting the audit data required, rather than enabling every available check by default.
5. Run a sample before the full segment. A sample exposes infinite pagination, client-side rendering, redirect loops, or filter expansion before they consume days of crawl time.

Google defines crawl budget as the resources Google can and wants to devote to crawling a site. Google’s guidance emphasizes controlling URL inventory and reducing unnecessary duplicate URLs. That is related to local crawler capacity, but it is not the same thing: running a desktop crawl does not itself improve Google crawl budget. The audit instead helps reveal duplicate paths, poor internal linking, and low-value URL patterns that may affect Googlebot’s discovery work.

The following sampling figures are author heuristics, not vendor limits or Google recommendations: an ecommerce audit could begin with 5,000 products, 2,000 category pages, 1,000 editorial URLs, and all indexable pagination patterns. The right number varies with template diversity. A catalog with 20 product templates needs broader sampling than one where every product page uses the same rendering and metadata rules.

Audra vs Screaming Frog vs Sitebulb for local audit work

Audra, Screaming Frog SEO Spider, and Sitebulb can all support SEO audit workflows, but published documentation supports different conclusions about each. There is no independent benchmark in the supplied sources that establishes one as universally “best.” The appropriate choice depends on audit scope, required data, available hardware, reporting needs, and workflow preference.

Audra: scoped, local-first audit reporting

According to Audra’s product description, Audra is a local-first desktop website auditing app that combines AI answer-engine visibility checks with technical SEO, performance, accessibility, best-practice, and link audits. It is positioned for SEOs, agencies, marketers, and site owners who need client-ready reporting without a subscription.

For a large site, that positioning suggests a sensible use case: separate audits for /products/, /collections/, /blog/, and major regional folders, then compare recurring template-level issues. For example, a team could establish whether slow category templates, missing metadata, accessibility failures, or weak answer-engine visibility repeat across high-value sections.

Audra’s public product brief does not establish a maximum URL capacity, supported authentication methods, staging-crawl capabilities, operating-system availability, or a specific licence price. Those should be confirmed directly before a large engagement. It should also not be assumed that a reporting-oriented audit tool is intended to retain a complete, continuously refreshed archive of 20 million raw HTML documents.

For agencies where answer-engine visibility is part of client reporting, an AEO audit workflow for client websites provides related context on assessing AI-answer visibility alongside conventional site checks.

Screaming Frog: configurable, database-backed crawling

Screaming Frog’s published guide offers the clearest vendor-documented large-crawl configuration details in this comparison. Database storage writes crawl data to disk rather than relying primarily on RAM, making it the relevant mode for large technical investigations. The vendor recommends using an internal SSD and provides approximate URL-capacity examples rather than guarantees.

The documented examples are useful for planning: approximately 2 million URLs with 4 GB allocated in database storage, about 5 million with 8 GB, and around 10 million with 16 GB and a 500 GB SSD. Resource settings, JavaScript rendering, available disk space, and page complexity can reduce or alter those figures.

Screaming Frog is particularly relevant where the audit question requires detailed URL rules, custom extraction, or exports, such as:

• Finding pages with conflicting canonical and robots directives.
• Identifying paginated URLs with broken hreflang references.
• Extracting a specific structured-data field across a defined URL list.
• Comparing canonical targets against internal-link destinations.

Its free edition has a 500-URL crawl limit. Current annual licence pricing is not stated here because it should be verified on Screaming Frog’s own pricing information at the time of purchase rather than treated as a fixed September 2026 fact.

Sitebulb: sample-first audit planning

Sitebulb’s supplied large-website support guidance recommends setting up a project, conducting a sample audit with limits, and choosing only the audit data needed. That is a valuable workflow regardless of tool because heavy audits and browser rendering can materially change crawl duration and machine load.

The supplied Sitebulb source does not substantiate a 500,000-URL desktop limit, cloud crawling for millions of URLs, a specific resource-exclusion setting, or current subscription details. Those claims should not be used as decision criteria without checking Sitebulb’s current product documentation.

Sitebulb may be considered by teams that value guided audit outputs and visual interpretation, but that is a workflow preference rather than evidence that it will outperform another crawler on every large site. Run the vendor-recommended sample against a representative directory before committing to a full crawl or a commercial plan.

Storage, memory, and crawl queues

A crawl can run out of memory because it stores far more than URLs: HTTP responses, titles, directives, link relationships, rendered-page data, extracted fields, and sometimes page resources. One million URLs with extensive internal linking is not one million isolated records.

For local work, database-backed storage on a fast SSD can reduce RAM pressure. Screaming Frog explicitly recommends database storage for larger sites. That shifts the constraint toward disk speed and available storage; an HDD may function, but it can become a practical bottleneck.

To reduce local-memory failures:

• Use database storage where the chosen crawler supports it.
• Reserve substantial free SSD space before the crawl begins.
• Disable checks, resources, and external URLs that do not answer the audit question.
• Test JavaScript rendering on a representative sample before enabling it site-wide.
• Crawl one directory, subdomain, or URL list at a time and archive completed segments.

A distributed crawler uses a different architecture. For example, a Crawlbase-style large-scale collection workflow can place discovered URLs in a central crawl queue; workers fetch URLs; a deduplication layer prevents repeat requests; retry logic handles temporary errors; and a database or object store retains results. The system-design concepts of politeness, queue management, deduplication, fault tolerance, and monitoring matter because workers can otherwise multiply origin load quickly.

A concrete decision threshold can be stated as an author heuristic: if the requirement is to collect and retain every URL response repeatedly, across several million URLs, with custom fetch logic and scheduled recrawls, a queue-based platform or engineering build should be evaluated. If the deliverable is a one-off or periodic SEO report on templates and priority sections, segmented local crawling is often simpler to operate and interpret. Neither path is automatically superior.

How to crawl a large website without getting blocked

An authorized SEO crawl should behave like a careful visitor, not an attempt to defeat access controls. Respect robots.txt, follow explicit client restrictions, identify the crawler where possible, and coordinate with the site owner before high-volume work.

Rate limiting is the key operational control. At five URLs per second, one million URLs require more than 55 hours of request time before redirects, rendering, retries, errors, and crawl delays. That arithmetic explains why increasing concurrency is not a harmless shortcut.

The following are author operating heuristics, not universal vendor settings:

• Start at one or two URLs per second on an unfamiliar origin.
• Watch response time, 429 responses, 403 responses, and 5xx errors during the first few thousand URLs.
• Ask hosting, CDN, and security teams to whitelist an approved audit where appropriate.
• Pause and investigate if error rates increase instead of raising concurrency.
• Agree a scope and authorization process before crawling a private or pre-launch environment.

Proxy rotation appears in scraping infrastructure because it can distribute traffic among IP addresses. It should not be a default tactic for SEO auditing or a means to evade a block. If the site blocks an audit, the appropriate response is to reduce request rate, check robots and firewall rules, or obtain explicit permission.

For teams planning authorized pre-launch reviews, this staging website crawling comparison discusses local audit workflow considerations. Tool support for a particular authentication method should still be verified directly with the vendor.

Segment by template, directory, and risk

Enterprise SEO guidance commonly recommends segmented crawling because representative sections can be analyzed without treating every URL as equally valuable. Segmentation reduces hardware risk, makes validation easier, and creates clearer remediation ownership.

A segmentation map can include:

• Product detail pages by catalog, brand, or template.
• Category and subcategory listing pages.
• Editorial, news, guide, and help-center sections.
• Country folders, language folders, and subdomains.
• High-revenue landing pages and recently redesigned templates.

The segment must be large enough to reveal a pattern. If 83 of 100 sampled category pages have duplicate titles, the likely remediation is a template or CMS-rule fix, not 83 manual edits. If canonical conflicts appear only in /de/, the issue may be isolated to international implementation.

Maintain a crawl log with seed URLs, inclusion and exclusion rules, date, request rate, storage mode, rendering choice, and known gaps. This matters when comparing a September crawl with a December crawl: without configuration records, a change in issue counts may only reflect a changed scope.

Segmentation can also support security review. If crawl findings show inconsistent browser protections on checkout or account-adjacent pages, a separate review of website security header configuration can help translate the finding into implementation work.

Manage Google crawl budget separately from audit scope

Crawl budget management is about helping search engines allocate crawling resources to URLs that matter; it is not about making a desktop crawler finish faster. Google says crawl-budget considerations are most relevant to very large sites, rapidly changing sites, or sites with substantial duplicate URL inventories.

A practical review should:

1. Identify canonical, indexable URLs worth regular crawling.
2. Find duplicate paths created by parameters, filters, calendars, internal search, and alternate sorting.
3. Remove unnecessary internal links to low-value variations.
4. Apply appropriate controls where crawling should be prevented.
5. Keep XML sitemaps focused on canonical URLs intended for discovery and indexing.
6. Compare crawler findings with Google Search Console and server logs before diagnosing a crawl-budget issue.

Faceted navigation is a recurring enterprise problem. Five filters with multiple values can create thousands of combinations, but some filtered category pages may serve distinct search demand. Google’s faceted-navigation guidance supports managing these URLs deliberately rather than applying a blanket rule to every parameter. The target is not merely fewer URLs; it is fewer low-value URLs competing with important pages for crawling and internal link equity.

Which should you choose for a large website crawl?

Choose Audra when the deliverable is an actionable, client-ready audit of priority site sections and the team wants the combined checks described in Audra’s product brief, including technical SEO, performance, accessibility, links, and AI answer-engine visibility. Confirm scale, platform, licence, and access requirements directly with Audra for the proposed scope.

Choose Screaming Frog SEO Spider when the work needs configurable URL rules, custom extraction, detailed exports, and vendor-documented database-storage planning into the millions of URLs. Plan SSD capacity, memory allocation, crawl settings, and operator time as part of the project cost.

Consider Sitebulb when its sample-first workflow, audit hints, and reporting style fit the team’s process. Use a representative test crawl and verify current limits, plans, and deployment options with Sitebulb rather than relying on unverified numerical claims.

Choose distributed infrastructure when continuous multi-million-URL collection, custom fetching, durable storage, a crawl queue, worker management, and engineering ownership are requirements. That is appropriate for a crawler product or data pipeline, but can be excessive for a conventional technical SEO engagement.

Verdict

The best way to crawl a large website is rarely to fetch every possible URL at maximum speed. Define the audit question, control URL noise, honor robots and rate limits, test a representative sample, and segment by template and directory.

Audra is positioned as a local-first, client-reporting audit option with AI visibility and conventional website checks. Screaming Frog provides the most specific supplied guidance for configured database-backed desktop scale. Sitebulb’s documented sample-audit approach promotes disciplined planning. Distributed crawlers are justified when the requirement becomes persistent, large-scale data collection rather than an interpretable SEO audit.

FAQ

How do you crawl a large website without running out of memory?

Use database-backed storage where the crawler supports it, keep the database on a fast SSD, and collect only data relevant to the audit. Screaming Frog recommends database storage for larger sites and estimates approximately 2 million URLs with 4 GB allocated. Segment by directory, exclude parameters, and test JavaScript rendering before enabling it across every URL.

What is the best tool for crawling a website with 1 million or more pages?

There is no single best tool without a defined output. Screaming Frog has vendor-documented database-storage examples into the millions, subject to SSD capacity and site complexity. A segmented audit tool can fit a client-reporting objective. A distributed platform fits recurring collection of every URL. Test a representative section and compare the required data, hardware, and reporting needs.

How can you crawl a large website without getting blocked?

Honor robots.txt, obtain authorization, begin at a conservative rate, and watch 429, 403, and 5xx responses. One or two URLs per second is a cautious author heuristic for an unfamiliar origin, not a universal setting. Coordinate with hosting or security teams for high-volume audits. Do not use proxy rotation to bypass a site’s access controls.

Should a large website be crawled in segments?

Usually, yes. Product, category, editorial, regional, and high-revenue sections are easier to validate and assign than one undifferentiated export. Segmentation can reveal template defects quickly: for example, duplicate titles on 83 of 100 sampled category pages point to a systemic implementation issue. Record each segment’s seeds, rules, speed, and rendering settings.

Can you crawl a large website for free?

A limited proof of concept can be free: Screaming Frog’s free edition is capped at 500 URLs. A large crawl still has operational costs, including SSD storage, machine capacity, operator time, server coordination, and potentially infrastructure engineering. Free software should not be confused with a cost-free enterprise crawl.