Grow · Intermediate
AI crawlers and robots.txt: allow or block, and what happens either way
A few lines in a text file decide whether AI assistants may read your website. Many companies have never looked at those lines, and some are blocking crawlers without knowing it.
Photo: Nilo Velez, public domain (CC0)
In this guide
Short answer
If you want AI assistants to name and cite your company, allow their crawlers; if you block them, you drop out of their answers. Crawlers that collect training data and crawlers that fetch pages for live answers have different names, so you can block the first kind and allow the second. robots.txt is a request, not a lock.
What is robots.txt, and what is it not?
It is a public text file with requests to crawlers, and nothing enforces it. The file sits at the top level of a website, at /robots.txt, and tells crawlers which addresses they should not fetch. Well-behaved crawlers read it first and follow it. A crawler that ignores it meets no technical barrier.
So robots.txt is no way to protect confidential pages: anyone can read the file, and it does not stop access. Whatever must stay private belongs behind a login.
Crawlers announce themselves with a name called the user agent, sent with every request. The rules in robots.txt are written per name.
Which AI crawlers are there?
There are two kinds, and the difference decides what a block costs you. Some crawlers collect pages that may be used to train models. Others fetch pages to answer a question someone is asking now, and those answers show your page as a source.
| Name in robots.txt | Operator | Purpose |
|---|---|---|
| GPTBot | OpenAI | Collects pages that may be used to train models |
| OAI-SearchBot | OpenAI | Finds pages for search answers in ChatGPT |
| ChatGPT-User | OpenAI | Fetches a page when a user’s request calls for it |
| ClaudeBot | Anthropic | Collects pages that may be used to train models |
| PerplexityBot | Perplexity | Finds pages for Perplexity’s search answers |
| Google-Extended | Not a crawler: a token that controls use for Google’s AI models | |
| Bingbot | Microsoft | The crawler of the Bing search index |
Google-Extended is a name you can use in robots.txt to say whether pages may be used for Google’s AI models. It does not affect whether a site appears in Google Search. Bingbot is in the list because it is an ordinary search crawler that is sometimes blocked by accident together with the others.
The list is not complete and it changes. Operators document their crawlers on their own websites, and that is the reliable place to look names up.
What do the rules look like?
A rule is a pair of lines: a name, then what that name should not fetch. Each line below is one line of the file.
The first example lets every crawler read the whole site except one folder:
User-agent: *
Disallow: /internal/
The second asks two training crawlers to stay away from the whole site and leaves every other crawler, the answering ones included, untouched:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
A crawler that is not named and finds no general rule may read everything. One detail causes mistakes: a crawler that finds a group with its own name follows only that group and ignores the general one under the star. If you add a group for one crawler, repeat in it whatever general rules should still apply to that crawler.
Should you allow or block AI crawlers?
For a company that sells something, allowing the answering crawlers is nearly always right, and the training crawlers are a matter of judgement. An assistant can only cite a page it was allowed to fetch.
| Choice | What happens | Who it suits |
|---|---|---|
| Allow everything | Pages can be cited in live answers and may be used in training | Companies whose content exists to win customers |
| Block training, allow answering | Pages can still be cited; you ask not to be used for training | Publishers and firms with costly original content |
| Block everything | You drop out of the answers of crawlers that obey; content already collected is not removed | Sites that are not meant to be found |
| Block single folders | The rest stays visible; the folders are not fetched | Any site with areas of no use to outsiders |
Where do blocks hide?
They hide in four places, and robots.txt is only the visible one. We have found websites that had blocked search and AI crawlers at server level for years without the owner knowing. Such a rule is typically added as a temporary measure and then forgotten.
- robots.txt: open it in a browser and read every line.
- A firewall or security plugin: many have a “block bots” switch or a list of names.
- The hosting provider: some block or slow down crawlers for all customers on a server.
- The .htaccess file or server configuration: rules that refuse requests by user agent.
How do you check your own site?
You request a page under the crawler’s name and look at the status code of the reply. A tool such as curl can send any user agent. For example:
curl -I -A “GPTBot” https://www.example.com/
- Read robots.txtOpen /robots.txt on every domain and subdomain you use. Each host has its own file.
- Request a page as a crawlerSend a request with the crawler’s name as user agent, once for each name that matters to you.
- Read the status code200 means the page was delivered. 403 means something refused the request. A challenge page or a timeout means a filter stepped in.
- Compare with a browserRequest the same page with an ordinary browser’s user agent. If that works and the crawler name fails, a rule by name is blocking.
- Look in the server logsSearch the access log for the crawler names. Real visits with status 200 are the best proof.
The test has a limit: it shows rules that go by the name only. A firewall may also judge by the address a request comes from, so the logs remain the final check. A block in robots.txt, by contrast, never shows in the status code. The server would deliver the page, and the crawler chooses not to ask.
How we do it
We read the robots.txt, test each host under the crawler names, and go through the firewall, hosting and server rules with whoever runs them. We remove blocks that nobody decided on, and write down the ones that are wanted and why. What the crawlers find once they are let in is the subject of search engine optimisation.
Some sites also publish a file called llms.txt, a plain summary of the site for AI systems. It is an emerging convention, not a standard, and no operator guarantees that it is read. Server rules and blocks are part of what we watch under managed hosting.
Questions we hear often
Want this sorted without doing it yourself?
Tell us what you are working on. A person replies within one working day.