Allowed
No rule in robots.txt keeps the crawler out of the site.
Free robots.txt checker for AI crawlers
Read a site's robots.txt and see which AI crawlers it allows and which it blocks.
- Free, with no signup
- Saves nothing it checks
What your result means
Partly blocked
The crawler may read the site except the paths listed for it.
Blocked
robots.txt asks the crawler to stay out of the whole site.
Questions about the robots.txt LLM policy
Do AI crawlers follow robots.txt?
The companies behind the crawlers listed here say they do. robots.txt is a request, not an access control, so it cannot stop a crawler that ignores it.
What is the difference between GPTBot and ChatGPT-User?
GPTBot collects pages for training. ChatGPT-User fetches a page when a person asks ChatGPT about it. Blocking one does not block the other.
What is Google-Extended?
A token, not a separate crawler. Blocking it tells Google not to use your pages for its AI models, and does not affect Google Search.
Found a problem? We will check the rest.
Book a deliverability audit and we will look at every domain and mailbox you send from.