Robots.txt & AI Bot Checker
See whether Googlebot, Bingbot, GPTBot, ClaudeBot, Google-Extended, and other crawlers are allowed to access a specific page.
Free, no signup. Rate-limited per visitor to keep this fast for everyone.
What this checker verifies
This tool fetches a site's robots.txt file and runs its rules against a specific page URL for each of the crawler user-agents listed above — Google's main crawler, Bing's, and the newer AI crawlers that feed ChatGPT, Perplexity and other AI assistants. If no robots.txt exists at all, everything is implicitly allowed by default, which this tool states plainly rather than guessing.
Why robots.txt matters
A single misplaced Disallow rule can accidentally block an entire section of a site from Google without anyone noticing — the page still loads fine for a human visitor, so nothing looks broken until rankings quietly disappear. This is one of the most common, and most avoidable, causes of a page never showing up in search at all.
Why AI crawlers are checked separately
Googlebot and Bingbot decide what shows up in search results; GPTBot, ClaudeBot, Google-Extended and similar crawlers decide whether an AI assistant can read a page at all when answering a question. A site can intentionally allow one and block the other — for example, staying in Google search while opting out of AI training — but that only works if the robots.txt rules actually say what the owner thinks they say.
What to do with the results
If a bot you want visiting your site shows as Blocked, check robots.txt for a rule matching that page's path and adjust it. If you want the same check run across every page on your site instead of one at a time, a free account covers up to 100 URLs per crawl.