robots.txt, llms.txt and AI bots: what to allow and what to block

The right choice depends on your goal. To appear in AI search answers, allow the search bots. If your materials should not be used to train models, block the training bots. Service sections – the cart, the customer account, internal search – are closed to everyone. The llms.txt file remains a voluntary community proposal, and its support depends on the specific service.

In brief

• robots.txt is placed in the root of the domain; it contains rules for individual bots by name (User-agent).
• AI bots perform different roles: model training, indexing for search and requests made on a user's behalf. Rules are set separately for each role.
• robots.txt is a request to well-behaved bots and does not protect data, so private information is protected with authentication.
• llms.txt is an optional, voluntary hint; indexing and content quality take priority.
• Bot names and operator rules change, so check them against the documentation and verify them in the server logs.

Three roles of AI bots

A bot's name by itself tells you nothing about its role, so first group the bots by purpose. Operators describe it in their public documentation.

The names are taken from the operators' public documentation at the time this article was prepared; they change, so check the current lists before adding rules. Google-Extended is a token for robots.txt: according to Google, it controls the use of materials for some of the company's AI products and does not affect the site's presence in search.

To find out who exactly visits your site, open the server logs or hosting reports and look at the User-Agent values. Among the entries you may find bots that are missing from your list: check them in the operator's documentation before blocking them.

What to allow and what to block

The decision about training bots depends on the business. Companies that attract customers through search and brand recognition often keep everything that helps people find them open. Publishers and authors of original content restrict training more often. A single rule for all bots at once almost never fits, since their roles differ.

Example rules

An example for a site that wants to stay visible in search and asks bots not to use its materials for training. Substitute your own bot names and paths.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /search/

Sitemap: https://example.com/sitemap.xml

How to read such a file:

• Each group starts with a User-agent line with the bot's name, followed by Allow and Disallow rules.
• A bot follows the most specific group for its name, so GPTBot has its own group, and the rules of the User-agent: * group do not apply to it.
• The path in a rule is given from the root, and paths are case-sensitive.
• The Sitemap line gives the full address of the sitemap.
• The file applies only to the domain or subdomain it is hosted on, so each subdomain needs its own.

If you want to stay open to all bots, it is enough to block only the service sections:

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/

Sitemap: https://example.com/sitemap.xml

What llms.txt is and whether you need it

llms.txt is a community proposal to place a Markdown text file in the root of a site with a short description of the site and links to its most important pages. The idea is to make it easier for language models to find the essentials, particularly in documentation and help sections. It is an unofficial initiative without an approved standard. Support depends on the specific tool.

There are no public guarantees that popular search engines take this file into account, so it is treated as an optional addition after the basics. If you decide to create the file, follow a few simple rules:

• Briefly describe the site and list the main sections with one sentence about each.
• Link to canonical addresses that respond with code 200.
• Keep the file up to date after changes to the site structure.
• Do not add instructions for models or questionable claims: the file should be a truthful reference.
• Remember that llms.txt blocks nothing: robots.txt exists for restrictions.

For most small sites, the time is better spent on indexing, page structure and markup, which are described in the article How to prepare a website for AI search: structure, data, trust.

Server load and scrapers

Some bots send a lot of requests, and on weak hosting this is noticeable. Check in the logs which bots create the main load. Request rate limits at the CDN or server level, page caching and WAF rules help. Not all bots support the Crawl-delay directive, so you should not rely on it alone. Be careful with hard blocks by country or IP range: they can cut off useful bots and real customers.

How to check the rules without harming the site

1. Open the /robots.txt address in a browser and make sure the file responds with code 200 and contains the expected rules.
2. Check the file in Google Search Console: the robots.txt report and the inspection of individual URLs show whether pages are available to Googlebot.
3. Analyze the server logs: User-Agent, request frequency, visits to blocked sections.
4. Verify that bots are genuine. A declared User-Agent is easy to fake; operators usually publish IP address ranges or verification methods.
5. Review the settings of your CDN, WAF and security plugins: some have built-in rules for AI bots that may contradict robots.txt.
6. Repeat the check a few days later: bots cache the file and do not react to changes instantly.

Common mistakes

• A Disallow: / for all bots left over after testing: the site disappears from search.
• Blocking the CSS and JavaScript needed to render pages.
• Trying to remove a page from search through robots.txt. A crawl ban does not remove the URL from the index; for that, use noindex, which the bot must be able to read.
• Copying someone else's list of bots without checking the names and purposes.
• Relying on robots.txt to protect private data: the file is a request to bots and is available to anyone.
• Blocking all AI bots without a decision about visibility in answers.

Questions and answers

Should I block GPTBot?

It depends on your goal. According to the operator's documentation, GPTBot is related to model training, and the search bot has a separate name. So blocking GPTBot is a request not to use your pages for training, and by itself it does not block the search bot. The decision is made by the content owner, taking the business model into account.

Does blocking AI bots affect rankings in Google?

According to Google, the Google-Extended token does not affect the site's presence in search. Blocking Googlebot itself, on the contrary, removes pages from search. So before changing the rules, check which bot they apply to.

Do all bots follow robots.txt?

No. Well-behaved operators follow it. Third-party scrapers may ignore the file, so unwanted traffic is handled with request rate limits, WAF rules and mandatory authorization.

Does an online store need llms.txt?

It is optional. For a store, the product feed, Product markup, a fast site and clear policies matter more. If you have spare resources, the file can be added as a reference to the main sections.

How often should robots.txt be reviewed?

After every change to the site structure, release or migration, and also on a schedule once every few months. Operators update bot names and purposes, so check your list against their documentation.

Conclusion

You can manage AI bot access without complex tools: define your goal, group the bots by role, set the rules in robots.txt and verify them in the logs. The llms.txt file remains an optional addition to the main work on the site. Reviewing the rules after every major release protects you from accidental mistakes. The LightSEO team – an agency from Mukachevo that has been working on websites and SEO since 2015 – can help check your robots.txt and the overall technical picture. A good place to start is an SEO audit of your website, and for a new project – a turnkey corporate website.

Read also

• AI ad automation: Performance Max and Advantage+ without illusions
• AI in marketing for beginners: what is changing and what to learn
• AI tools for small business: what is worth trying