What this tool does
robots.txt still controls many AI crawlers. Blocking GPTBot or Google-Extended by accident can shrink your visibility in AI products that respect robots rules.
Why it matters
Teams often copy a “block all AI” snippet without realizing it also blocks useful browsing and grounding bots.
How it works
- We fetch /robots.txt and look for named AI user-agents.
- We draft an allow-oriented policy you can edit.
- Email unlocks the full draft.
Tips
- Review legal/policy before allowing training crawlers.
- Always keep a Sitemap: line.
- Re-scan after CDN changes.
FAQ
Should I allow every AI bot?
Not necessarily. Decide per bot: browsing vs training. This tool makes the current state visible.