免费 · SEO
AI 爬虫 robots.txt 生成器.
在 robots.txt 中屏蔽或允许 GPTBot、ClaudeBot、Google-Extended、PerplexityBot、CCBot 等 AI 爬虫：屏蔽训练、保留 AI 搜索或全部屏蔽。
Loading the calculator…
使用方法: AI 爬虫 robots.txt 生成器
输入数据或选择文件，调整设置并生成结果。复制或下载之前，请检查结果。
限制与支持格式
输入内容在您的设备上处理。
英语技术说明
robots.txt is voluntary; some crawlers ignore it. User-agent names change, so check each company’s documentation. Everything is calculated in your browser; nothing you type is uploaded.
Training crawlers collect pages for model training; search crawlers index pages for AI answers; user agents fetch a page when a person asks. Blocked crawlers get “Disallow: /”; the others follow your normal rules.
The default policy blocks the 7 training crawlers (GPTBot, ClaudeBot, Google-Extended…) and keeps the 7 search and user-request crawlers allowed.