AI Bot & Robots.txtAccess Checker

robots.txt controls crawling — whether a bot is even allowed to fetch a URL. The meta robots tag (and the X-Robots-Tag header) control indexing — whether a page Google fetched is allowed to appear in results. A page can be crawlable but set to noindex, or blocked in robots.txt yet still indexed from links. This tool shows you both gates for the URL you enter.

What's new is the cast of crawlers. Alongside Googlebot and Bingbot, your audience increasingly arrives through AI assistants — and OpenAI's GPTBot, Anthropic's ClaudeBot, PerplexityBot, Google-Extended and others each read your robots.txt to decide whether they may use your content. Accidentally blocking them (or deliberately wanting to) is now a real decision, and most site owners have never checked.

Green means that crawler is allowed to fetch the page. Red means your robots.txt blocks it. Neither is automatically "right" — but you should know which choice your site is making.