What is Crawl Budget?
Crawl budget is the number of pages Googlebot will crawl on your website within a given timeframe. It's determined by two factors: crawl rate limit (how fast Google can crawl without overloading your server) and crawl demand (how much Google wants to crawl your pages based on their popularity and freshness). Managing crawl budget is critical for large sites where not all pages get crawled and indexed in a reasonable time.
- Crawl budget = crawl rate limit × crawl demand — optimise both factors.
- Only matters when you have hundreds of pages or more; small sites are crawled fully by default.
- Blocking low-value pages (tag archives, filtered URLs) frees budget for pages that matter.
- Server response time under 200ms is the single most impactful crawl budget lever.
- Check your crawl stats in Google Search Console → Settings → Crawl Stats.
What Determines Your Crawl Budget?
Google allocates crawl budget based on two things: how fast your server can handle requests without slowing down, and how much value Google sees in your pages.
Crawl rate limit is Google's self-imposed throttle. If your server responds slowly, Googlebot backs off to avoid degrading your site's performance for real users. Faster servers = more crawling.
Crawl demand is driven by how popular your URLs are (backlinks, traffic) and how frequently they change. A homepage gets crawled daily. A deep archive page from 2012 might get crawled monthly — or never.
Signs You Have a Crawl Budget Problem
Most sites don't have a crawl budget problem. If you have fewer than a few hundred pages, Google will crawl all of them regularly.
You need to care about crawl budget if: new pages take weeks to appear in search results, you have thousands of URLs but low organic traffic, or Google Search Console shows a crawl anomaly.
The clearest signal is in Search Console's Crawl Stats report — compare daily crawl volume against your total page count. If Googlebot is visiting a fraction of your pages, you have a problem worth solving.
Most guides are already outdated.
One email a week. The search stuff that actually matters — what shifted, what died, and what to do about it.
Subscribe free →How to Optimise Crawl Budget
The fastest wins come from blocking what shouldn't be crawled. Use robots.txt to block tag archives, filtered product pages, and admin URLs. Add noindex to thin pages you can't delete.
Consolidate duplicate content with canonical tags. Fix redirect chains so Googlebot doesn't waste hops. Remove dead URLs from your sitemap — only include canonical, indexable pages.
On the server side: improve response time, reduce server errors, and ensure your most important pages are reachable within 3 internal links from the homepage.
| Crawl Budget Is Critical | Crawl Budget Is Not a Priority |
|---|---|
| 10,000+ indexable URLs | Fewer than ~1,000 pages |
| New pages take weeks to appear in Search Console | New pages indexed within 1–3 days |
| Large e-commerce with faceted navigation | Small business or blog site |
| News or content sites publishing dozens of articles daily | Static brochure site updated monthly |
| GSC Crawl Stats show only a fraction of pages crawled per day | GSC shows consistent daily crawl coverage |
| Multiple redirect chains and legacy URL structures | Clean, flat URL architecture |
Search engines including Google began honouring the crawl-delay directive in robots.txt, giving site owners a basic mechanism to throttle Googlebot and protect server performance.
Google's Gary Illyes published the first official blog post explaining crawl budget as the product of crawl rate limit and crawl demand, giving SEOs a formal framework for the first time.
Google launched the Crawl Stats report in the new Search Console, surfacing daily crawl data, response codes, and file-type breakdowns — making crawl budget analysis accessible without log file analysis.
Google clarified that JavaScript-heavy pages require a two-stage crawl-and-render process, effectively doubling the crawl budget cost of client-side rendered content versus server-side HTML.
How does your site score on SEO?
Paste your URL. Get a score and a fix list across all three disciplines. No form, no email.
Run Free Audit →Frequently Asked Questions
Not directly — crawl budget affects indexation, which affects whether your pages can rank at all. If a page isn't crawled, it can't be indexed, and if it isn't indexed, it can't rank. So indirectly, yes: poor crawl budget management can suppress rankings on large sites.
Go to Google Search Console → Settings (bottom left) → Crawl Stats. You'll see average daily crawl volume, response time breakdown, and crawl requests by response code. Compare the daily crawl number against your total indexed page count.
Only block pages you're confident should never be indexed. robots.txt prevents crawling but Google may still index a blocked URL if it has external links pointing to it. For pages you want deindexed, use noindex instead. Use robots.txt for pages that genuinely waste crawl budget — parameter URLs, faceted navigation, session IDs.
- 1.Google Search Central — Crawl Budget documentation
- 2.Screaming Frog SEO Spider documentation
- 3.Advanced Web Ranking CTR Study, 2024
Read next
robots.txt
A robots.txt file is a plain text file at the root of your domain that instructs web crawlers which pages or s…
XML Sitemap
An XML sitemap is a file that lists all the URLs on your website that you want search engines to crawl and ind…
Canonical Tags
A canonical tag (rel="canonical") is an HTML element that tells search engines which version of a URL is the '…
Programmatic SEO
Programmatic SEO is the practice of using data and templates to generate large numbers of unique, search-optim…