What Is Crawl Budget (and When It's Actually a Problem)

Ask “what is crawl budget” in most SEO circles and you will get a confident answer followed by some questionable advice. The term has become the default explanation for any page that has not appeared in Google yet, which is a shame, because crawl budget is a real constraint with a precise meaning, and for the majority of sites it is not the constraint that matters. This post defines the term properly, quotes the thresholds Google itself publishes, and gives you a way to tell whether crawl budget explains your situation or merely decorates it.

What is crawl budget?

Crawl budget is the amount of attention Googlebot is prepared to spend on your site: how many URLs it will fetch, and how often. Google describes it as the product of two forces.

The first is the crawl capacity limit. Googlebot does not want to knock your server over, so it watches how your site responds. Fast, healthy responses and the limit drifts up. Slow pages, timeouts, and server errors and it drifts down. This half is about what your infrastructure can absorb.

The second is crawl demand. Google crawls what it believes is worth crawling. Popular pages, pages that change often, and pages it has not seen before generate demand. Pages that never change, duplicate other pages, or sit unlinked at the bottom of a URL scheme generate very little. This half is about what Google thinks your URLs deserve.

Your effective budget is whichever of the two runs out first. A fast site full of pages nobody links to has capacity without demand. A popular site on a struggling server has demand without capacity. Neither gets crawled the way its owner would like.

Google’s own thresholds

Here is the part that rarely makes it into the confident answers. Google’s crawl budget documentation opens by saying who should read it, and the bar is high: sites with more than 1,000,000 unique pages whose content changes weekly, or sites with more than 10,000 unique pages whose content changes daily. Google is explicit that smaller sites generally do not need to worry, and that publishing pages the same day they are crawled matters mainly for genuinely time-sensitive content.

Read those numbers against your own site. A 300-page company site is not fighting for crawl budget. A 4,000-URL shop that updates weekly is not either. If a page there sits unindexed, the cause lives elsewhere: a noindex directive, a canonical pointing away, thin content, or simply a judgement Google has not made yet. The reasons Google declines to index a page that looks fine are almost always more productive suspects than crawl budget on a site of that size.

The symptom that actually points to crawl budget

There is one signal Google names directly: a large share of URLs sitting in Discovered - currently not indexed. That status means Google knows the URL exists and has chosen not to fetch it yet, which is precisely what a budget shortfall looks like from the outside. If you run a large site and thousands of URLs sit in that state for weeks, crawl budget deserves a place on your list. On a small site, the same status usually means something less dramatic, and the causes and fixes for Discovered - currently not indexed are worth walking through before reaching for infrastructure work.

The other honest signals are relative, not absolute. New pages that used to be crawled within days now wait weeks. The Crawl Stats report shows falling requests alongside rising response times. Sections of the site with known changes are being refetched rarely or not at all. One slow week proves nothing; a trend across months on a large site is worth acting on.

What actually moves the budget

If you clear the bar, the levers are unglamorous and mostly subtractive.

Remove the URLs that waste the budget. Faceted navigation, session parameters, infinite calendars, and printer-friendly duplicates can multiply a 50,000-page site into millions of crawlable URLs. Every fetch spent on ?sort=price&color=blue is a fetch not spent on a page you care about. Consolidate duplicates with canonicals, block genuinely useless spaces in robots.txt, and stop generating URLs you never intended to exist.

Fix soft errors. Pages that return 200 with no real content, and error pages that fail to say so, keep getting recrawled because Google cannot tell they are dead. Returning proper 404s for gone pages lets the budget flow elsewhere.

Keep your sitemaps honest. A sitemap with accurate lastmod values is the cheapest way to tell Google where the changes are, so demand lands on the right URLs. A sitemap that stamps today’s date on everything teaches Google to ignore the field.

Make the site faster. The capacity half of the equation responds to response times. Faster pages raise the ceiling on what Googlebot will attempt.

Knowing whether any of it worked

Crawl budget work has a measurement problem: the crawl is invisible from the outside, and Search Console reports arrive aggregated and delayed. What you can observe precisely is the outcome, per URL. This is where the distinction between submission and indexing earns its keep. Submitting a URL asks Google to look; only the index itself tells you whether Google did. AIndexify checks each submitted URL against live Google results on a fixed schedule and records the date a page was first found, so a batch of, say, 5,000 programmatic URLs submitted by CSV becomes a dated record of which ones made it in and which still have not. Run that before and after a cleanup and you have evidence, rather than a feeling, that the crawl is reaching more of your site.

For the wider context on how crawling, indexing, and serving fit together, the guide to Google indexing covers the whole pipeline end to end.

The short version: crawl budget is real, Google publishes the thresholds, and most sites are nowhere near them. Diagnose before you optimise. If you want to know where your pages actually stand while you do, submit your URLs and watch the indexed dates come back, or not, on a schedule you can hold them to.