AI Crawler robots.txt Generator
Choose which search and AI crawlers to block, add your sitemap and any private paths, and copy a ready robots.txt file. Everything runs in your browser.
FreeNo sign-upRuns in your browser
Tick a crawler to block it. Unticked crawlers stay allowed under the default rules.
How do you use the AI Crawler robots.txt Generator?
- Tick each crawler you want to block; leave the rest unticked to allow them.
- Add your sitemap URL and any paths that no crawler should visit.
- Copy the result and save it as robots.txt at the root of your domain.
Does robots.txt stop AI companies from using my content?
No file can force that. Google's robots.txt guide (checked 12 November 2026) says the rules cannot enforce crawler behaviour and that it is up to each crawler to obey them, so it is a request, not access control.
Which crawler names did we include?
Each token comes from the vendor's own documentation, for example OpenAI, Anthropic, Perplexity and Google (checked 12 November 2026). Bytespider is left out because we could not find official ByteDance documentation for it.
What else do people ask about the AI Crawler robots.txt Generator?
Will blocking Google-Extended remove my site from Google Search?
No. Google says Google-Extended only controls use of content for Gemini training and grounding, and it does not affect inclusion or ranking in Google Search. Googlebot controls Search.
What is the difference between GPTBot and OAI-SearchBot?
OpenAI uses GPTBot for crawling that may train its models and OAI-SearchBot for showing sites in ChatGPT search. You can block one and allow the other.
Do user-triggered bots follow robots.txt?
Not always. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person asked for the visit.
Is robots.txt a security measure?
No. It is a public request that well-behaved crawlers follow. Use passwords or server rules for content that must stay private.