مولد robots.txt لزواحف الذكاء الاصطناعي.

robots.txt is voluntary; some crawlers ignore it. User-agent names change, so check each company’s documentation. Everything is calculated in your browser; nothing you type is uploaded.

Training crawlers collect pages for model training; search crawlers index pages for AI answers; user agents fetch a page when a person asks. Blocked crawlers get “Disallow: /”; the others follow your normal rules.

The default policy blocks the 7 training crawlers (GPTBot, ClaudeBot, Google-Extended…) and keeps the 7 search and user-request crawlers allowed.