Know What Search Engines Actually Do On Your Site Before It Costs You Traffic

It tells you what search engines and AI tools are actually doing on your WordPress site, so you can stop guessing whether your content is being found. You’ll see which pages bots crawl and which they ignore, what errors they hit, where your crawl budget is being wasted, and whether AI assistants like ChatGPT and Claude are reaching your content. Most importantly, it tells you what to do about anything that’s wrong — including auto-generated robots.txt suggestions and one-click Google URL inspections.

Search Console only tells you about Googlebot, with a delay of two to three days. We show you every bot that visits your site — Googlebot, Bingbot, ChatGPT, Claude, Perplexity, Grok, and many more — in real time. We also tell you which of those visits are real and which are impostors, where your crawl budget is being wasted, and what to fix first. We don’t replace Search Console; we complement it. In fact, we integrate with it: you can run Google’s URL Inspection on any of your pages from inside our dashboard with one click.

No. Traditional server log analysis needs SSH access and the ability to parse raw access logs — which most WordPress users don’t have, and don’t want. Our tool is a regular WordPress plugin. You install it, activate it, and it starts capturing crawler activity at the WordPress application level. It works on shared hosting, managed WordPress, VPS, or dedicated servers. No FTP, no log files, no command line.

No. The plugin is built so visitors and bots don’t notice a thing. For the technically curious: each bot visit triggers a single, lightweight database insert. The heavy work — health scores, crawl-waste analysis, bot verification — runs only when you open the dashboard, with results cached anywhere from 5 minutes to 30 days depending on the operation. DNS-based bot verification has a 3-second time budget per dashboard load, with 30-day result caching, so it never holds anything up.

All the major ones, broken down by purpose. OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User. Anthropic: ClaudeBot, Claude-User, Anthropic-AI. Google: Google-Extended (Gemini training), plus Googlebot split into mobile and desktop. Apple: Applebot, Applebot-Extended. Perplexity: PerplexityBot, Perplexity-User. Plus xAI Grok, MistralAI-User, DuckAssistBot, Meta-ExternalAgent, ByteDance Bytespider, Common Crawl CCBot, Amazonbot, Cohere-AI, AI2Bot, Diffbot, YouBot, and more. New crawlers are added on plugin updates and tracked automatically.

Anyone can put “Googlebot” in the user-agent header of an HTTP request. To know whether a hit is actually Google, you have to verify the source IP — Google publishes the procedure: do a reverse-DNS lookup on the IP, check it ends in .googlebot.com or .google.com, then resolve that hostname forward and check it points back to the same IP. We do this automatically for the major bots that publish such procedures (Google, Bing, OpenAI, Anthropic, Apple, Perplexity, Yandex, Baidu, DuckDuckGo, Meta, Amazon), cache the result for 30 days, and show you a per-bot verified-vs-spoofed split with sample impostor IPs.

Search engines have a finite amount of time they’ll spend on your site. If they spend it crawling URLs that don’t earn you anything in search, that’s wasted. We categorise every bot visit into 16 buckets across three tiers: hard waste (server errors, broken links, redirect loops), probable waste (URLs with tracking parameters, internal site search results, faceted-navigation explosions, comment-pagination URLs, low-value tag/author/date archives, REST API hits, media files, soft-404 candidates) and configuration waste (the same content reachable at multiple URLs, like trailing-slash duplicates). Each row tells you the count, share of total crawls, sample URLs, the trend versus last week, and exactly what to change. The killer move: we read your live robots.txt, identify which fixes you haven’t already applied, and generate a copy-pasteable robots.txt block.

Connect Linkilo to Google Search Console once via Google’s OAuth flow. After that, every URL row in the dashboard has an Inspect button. Click it and we call Google’s URL Inspection API, then show you the same data Search Console displays: indexing verdict (PASS / PARTIAL / FAIL), coverage state, robots.txt status, page-fetch state, last Google crawl time, and both your declared canonical and the canonical Google actually chose. There’s also a Bulk Inspect button that processes all visible at-risk pages with a progress bar — throttled below Google’s 600-requests-per-minute limit so you never get cut off. Tokens auto-refresh transparently when they expire, so you don’t have to babysit the connection. Results cache for seven days, so re-clicking Inspect on a recently-checked URL is instant and doesn’t burn API quota.

It’s a single 0–100 number averaged from four equally-weighted components. Crawl Frequency: how often Googlebot visits, measured over the past week. Coverage: what share of your published content (across every public custom post type) Googlebot has actually crawled in the past 30 days. Error Rate: how many crawl requests returned 4xx or 5xx, with server errors weighted twice as heavily as client errors. Response Time: how fast your server replies, scored using the 95th percentile of successful responses (so a single slow request can’t tank the score). Each component shows its own status, the underlying numbers, and specific recommendations.

Yes. The Reports tab is built for agencies and in-house teams who have to show somebody the work is paying off. Save a snapshot of your crawl health (you can label it, e.g. “May client report”) and it captures every key metric at that moment in time. Open any saved snapshot to view a printable report — your browser’s Print → Save as PDF gives you the deliverable. Pick a second snapshot from the dropdown to add a comparison column with green/red change arrows on every metric. CSV export is available for any snapshot, with multi-section data ready for spreadsheet analysis. Optional: enable weekly auto-snapshots and Linkilo will quietly capture data every Monday in the background, so you always have a year of comparable history when reporting time comes.

Four types: No Crawl Activity (alerts when a chosen bot like Googlebot stops visiting for a configurable number of hours), High Error Rate (when error percentage exceeds your threshold), Low Coverage (when bot coverage drops below your target), and Slow Response Time. Alerts run on an hourly schedule. The Bot Behavior dashboard also includes automatic anomaly detection that highlights any bot showing a 3×-or-greater spike, or a 4×-or-greater drop, versus its rolling seven-day baseline.

We auto-detect your sitemap whether it’s generated by Yoast SEO, Rank Math, AIOSEO, or WordPress core, follow nested sub-sitemaps, then compare the URL list to what Googlebot has actually crawled in the past 30 days. We surface two lists: URLs your sitemap claims matter that Google hasn’t fetched (often thin content or low-link-equity pages worth removing or strengthening), and content URLs Google has crawled that aren’t in your sitemap (real pages your sitemap is missing). It’s a quick way to spot two opposite kinds of sitemap problems in the same view.

Logging starts the moment you activate the plugin — you’ll see bot visits within minutes. Most sites have enough data for meaningful health scores, crawl-waste analysis, and AI bot insights within 24 to 48 hours. Higher-traffic sites see useful data even faster. The verified-bot check populates lazily as you open the dashboard, with results cached for 30 days, so subsequent loads are instant.

Yes. Most dashboards have a CSV export — full crawl logs, the at-risk pages list, the overview stats, the report snapshots. Drop them straight into Google Sheets, Excel, or any BI tool. For agencies running custom analysis the snapshot CSV is multi-section: summary, status codes, bot mix, top URLs, coverage by bot, health breakdown, crawl waste headline.