Crawl budget: what it is and who should think about it
LightSEO · Technical SEO
Crawl budget: what it is and who should think about it
A search robot has a limited budget for crawling your site. If it spends that budget on junk pages, it simply never gets to your new products.
10,000+
pages – when it matters
The gist
What it means in practice
The robot does not crawl the whole site every day. It decides how many pages to check based on the site's size, speed and quality.
For small sites this is no problem. If you have two hundred pages, the robot will crawl them all. The question comes up at thousands of URLs.
A sign of trouble. New products take a long time to appear in search, while the index holds lots of junk URLs.
Leaks
What the budget is wasted on
🎚
Filter combinations
Thousands of URLs nobody searches for.
🔁
Redirect chains
Every hop is a separate robot request.
🕳
Errors and duplicates
Pages that return 404 or duplicate each other.
Solutions
How to save the budget
Close filters and sorting. The basic rule: only combinations opened on purpose get indexed.
Remove redirect chains. A direct link instead of three hops.
Speed up the server. The faster the site responds, the more pages the robot manages to crawl.
Watch the crawl stats report. Search Console shows how many requests the robot makes and what exactly it downloads.
New products take too long to show up in search?
We will check the crawl stats and remove whatever takes the budget away from your important pages.
