FREE · SEO TOOLS
robots.txt Generator.
Generate crawl directives and an optional sitemap reference.
Steps to use robots.txt Generator
Enter or choose the required input, adjust the settings, then generate and review the result. Allow crawling and add a sitemap URL.
How robots.txt Generator processes your input
Generate crawl directives and an optional sitemap reference. robots.txt controls crawling, not privacy or guaranteed deindexing. Inputs stay on your device. Review the preview or output before using it; refresh the page to clear the current working session.
robots.txt Generator: worked example
Allow crawling and add a sitemap URL.
How to create a robots.txt file
- Keep “User agent” as
*to address every crawler, or type one name such asGooglebot. - In the text box, list the paths to block, one per line, each starting with
/, for example/private/or/cart/. Leave it empty to allow everything. - Choose “Disallow listed paths” or “Allow listed paths”, add your “Sitemap URL (optional)” and pick an “AI crawlers” option.
- Copy or download the result and upload it as
robots.txtat the root of your domain, for examplehttps://example.com/robots.txt.
With /private/ and a sitemap address, the file reads User-agent: *, Disallow: /private/ and Sitemap: https://example.com/sitemap.xml. With no path, the tool writes an empty Disallow: line, which allows the whole site. Paths are matched from the start of the URL, so /private/ also covers /private/files/report.pdf.
AI crawlers, and what robots.txt cannot do
“Block AI training crawlers (keep AI search)” adds rules for GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider and Meta-ExternalAgent while leaving AI search bots alone. “Block all AI crawlers” also blocks OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User and Amazonbot, which can remove your pages from AI answers. For more control over each bot, use the AI robots.txt generator.
robots.txt controls crawling, not indexing or privacy. A blocked page can still appear in search results if other sites link to it, and anyone can read the file, so never list secret folders in it. To keep a page out of Google, let it be crawled and add a noindex meta tag, for example with the Meta Tags Generator. To create the sitemap you reference, use the XML Sitemap Generator.
Questions about robots.txt Generator
Where do I put the robots.txt file?
At the root of the domain, as https://example.com/robots.txt. Crawlers do not look for it in subfolders, and each subdomain needs its own file.
Does Disallow remove a page from Google?
No. It stops crawling, but a blocked URL can still be indexed from links. Use a noindex tag on a crawlable page to remove it from results.
Do AI crawlers obey robots.txt?
The major AI companies say they honour it, but compliance is voluntary and some crawlers have been reported to ignore it. The generated file says so in a comment.
Is robots.txt Generator free, and do I need an account?
Yes. This tool is free to use in your browser without an account. Processing takes place on your device.
What are the limits and what should I check?
robots.txt controls crawling, not privacy or guaranteed deindexing. Review the result before using it in your work.
Is my input sent to a server?
Your input is processed in your browser and is not uploaded by this tool. Files and pasted text are cleared when the page is refreshed. Normal website requests are still required to load the page and its libraries. Read the privacy explanation.