FREE · SEO TOOLS
AI Crawler robots.txt Generator.
Block or allow AI crawlers such as GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot in robots.txt: block training while keeping AI search, block everything, or choose crawler by crawler.
Loading the calculator…
How to use AI Crawler robots.txt Generator
Choose a policy or pick each crawler, add the paths that should stay closed to all robots and your sitemap URL, then copy the robots.txt.
How it works
Training crawlers collect pages for model training; search crawlers index pages for AI answers; user agents fetch a page when a person asks. Blocked crawlers get “Disallow: /”; the others follow your normal rules.
Example
The default policy blocks the 7 training crawlers (GPTBot, ClaudeBot, Google-Extended…) and keeps the 7 search and user-request crawlers allowed.
Questions about AI Crawler robots.txt Generator
Is AI Crawler robots.txt Generator free, and do I need an account?
Yes. This tool is free to use in your browser without an account. Processing takes place on your device.
What are the limits and what should I check?
robots.txt is voluntary; some crawlers ignore it. User-agent names change, so check each company’s documentation. Everything is calculated in your browser; nothing you type is uploaded. Review the result before using it in your work.
Is my input sent to a server?
Your input is processed in your browser and is not uploaded by this tool. Files and pasted text are cleared when the page is refreshed. Normal website requests are still required to load the page and its libraries. Read the privacy explanation.