AI Bot & Robots.txtAccess Checker
Is your page open to Google — and to ChatGPT, Claude and Perplexity? Check your robots.txt and indexing tags in one click and see exactly which crawlers you allow or block.
We read the site's robots.txt and the page's indexing tags server-side. Crawl rules are evaluated for the path you entered.
Two different gates — both matter now
robots.txt controls crawling — whether a bot is even allowed to fetch a URL. The meta robots tag (and the X-Robots-Tag header) control indexing — whether a page Google fetched is allowed to appear in results. A page can be crawlable but set to noindex, or blocked in robots.txt yet still indexed from links. This tool shows you both gates for the URL you enter.
What's new is the cast of crawlers. Alongside Googlebot and Bingbot, your audience increasingly arrives through AI assistants — and OpenAI's GPTBot, Anthropic's ClaudeBot, PerplexityBot, Google-Extended and others each read your robots.txt to decide whether they may use your content. Accidentally blocking them (or deliberately wanting to) is now a real decision, and most site owners have never checked.
Green means that crawler is allowed to fetch the page. Red means your robots.txt blocks it. Neither is automatically "right" — but you should know which choice your site is making.
The Linkilo features behind this tool
This checks one page. Linkilo tracks what bots actually do across your whole site, in real time.
Crawler & robots.txt questions
Nice. Now stop doing this by hand.
See exactly which pages Google and the AI crawlers actually visit, inside WordPress, with no log files to download.
30-day money-back guarantee. Every feature is included in every plan.
Orphan Pages Tool & Reporting
Find the pages nothing links to, the ones Google struggles to find, and get them connected.
Broken Link Checker & One-Click Repair
Find every dead link on your site and fix it in one click. No spreadsheets, no manual editing.
Redirection Manager
Catch 404s as they happen and redirect them before anyone notices the traffic went missing.
AI Bot & Robots.txt Checker (free)
Check which AI crawlers your robots.txt actually lets in, and which ones it blocks.