Robots.txt checker
Is your robots.txt blocking AI?
Enter your domain. We fetch your robots.txt and test it against 12 AI crawlers, your sitemap and the mistakes that hide a whole site.
No sign-upResults in secondsNothing stored
Works on any public domain, yours or a competitor's. We read /robots.txt exactly as a crawler would.
How to use it
One file. Every crawler’s answer.
robots.txt decides which engines can read your site. It takes a minute to make sure it says what you mean.
- 1
Enter your domain
Just the domain is enough. We fetch the live file from the root of your site, exactly as a crawler would.
- 2
Read the findings
Every problem is listed worst first, with the exact rule that causes it and the crawlers it affects.
- 3
Fix it and run it again
Change the rule in your CMS or on your server, then check again until the crawlers you want are allowed.
FAQ
Questions, answered.
Whether the file exists and loads, whether a blanket Disallow: / hides your whole site, whether any of 12 AI crawlers (GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended and others) are blocked, whether a sitemap is declared, and two quieter problems: long crawl delays and rules that block every URL with a query string.
Citeca
Put your SEO
on autopilot.
Citeca publishes articles on your site, drafts the fixes this tool finds, and shows which AI engines name you. 7 days free, then $99 a month.