robots.txt for an online store: what to block and what to keep open
LightSEO · Technical SEO
robots.txt for an online store: what to block and what to keep open
A single line in robots.txt can block the entire website from indexing. And it happens regularly – especially after moving from a test server to the production one.
5
typical sections to block
What it is
How robots.txt works
It is a recommendation with no power to enforce it. The file tells crawlers where not to go. Yet a page blocked in robots can still get into the index through links.
It does not hide a page from people. Robots.txt is available to everyone at the website address – do not list internal paths in it.
Reliable blocking requires noindex. Or a server-level restriction if the page really must not be public.
Block
What usually does not belong in the index
🛒
Cart and checkout
Personal pages with no value for search.
👤
Customer account
Everything behind a login.
🔎
On-site search results
They generate an endless number of URLs.
⚙️
Internal sections
Admin panel, system files, test pages.
Mistakes
The most common failures
The whole site blocked after development. The line that blocks the entire website was left in place after the move to the production domain.
Blocked styles and scripts. The search engine cannot render the page and sees it as broken.
Trying to block duplicates through robots. This does not work: duplicates need canonical or noindex.
Not sure what is in your robots.txt?
We will check the file and your indexing settings – and show what is blocked for no reason and what, on the contrary, should be blocked.
