Linkilo records every visit from Googlebot, Bingbot, GPTBot, ClaudeBot and other crawlers and shows you which pages they actually fetch, which they ignore, and which "bots" are fakes. In Linkilo 3.0 the report is called Crawler Analyzer.
What this is
Crawl analysis is what big SEO teams use log-file tools for. Linkilo brings that capability inside WordPress, focused on the data that matters: which posts crawlers care about, which they ignore, and which fake bots are pretending to be Googlebot.
Linkilo records the visits itself as bots request your pages, so you don't need access to your host's raw server logs. The feature is part of Linkilo — there is no separate add-on to install.
Turn it on
- Go to Linkilo → Manage → Settings → Server Log Analyzer (under the Tools group).
Two cards.
Card 1: Who's crawling your site
- Record crawler visits — master toggle. When OFF, nothing is recorded and the Crawler Analyzer report disappears from the menu.
- Keep visit history for — 7 days / 30 days / 90 days / 200 days (recommended). The "Days since last crawl" view needs at least 200 days of history to be useful.
Card 2: Pick which bots to record
By default, everything is on. Use the buttons for quick selection:
- Select all / Deselect all
- Search engines only — Googlebot, Bingbot, YandexBot, Baiduspider, etc.
- AI crawlers only — GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, etc.
- Search engines + AI — both groups at once
Below the buttons are individual checkboxes grouped into categories. Every bot you select gets its own line in the Crawler Analyzer charts when it has activity; anything Linkilo can't identify is grouped as Unrecognized Bots/Crawlers.
Click Save Settings.
Open the report
Go to Linkilo → Fix → Crawler Analyzer.
At the top you'll see four cards — Health, Issues, Uncrawled and Errors — and a date range and bot selector that apply to everything below.
Next Move
Right under the cards, the Next Move panel reads the loaded crawl data and points you to the fastest cleanup: crawler errors, slow high-crawl URLs, redirect waste, or pages no bot has visited. One button takes you to the matching tab, so you don't have to interpret raw logs first.
Tabs
- Overview — Crawl Activity chart (one line per bot), Status Codes, and Most Crawled URLs (top 20 in the selected window) with a crawl-ratio chip showing whether a URL is hot, warm or cool compared with your other busy pages.
- Health — the crawler health score and what's dragging it down.
- Coverage — what share of your pages each bot has fetched, plus Recently Updated Pages Needing Re-Crawl (pages edited in the last 60 days that the bot hasn't fetched since the edit).
- Days Since Crawl — Pages by Days Since Last Crawl for the selected bot and risk level, with badges for pages blocked by robots.txt or set to noindex, and a GSC Insights card that aggregates Google's indexing verdicts and can bulk-inspect the pages shown.
- Crawl Budget & Waste — Crawl Waste Analysis: errors, 404s, redirect chains, parameter URLs and other requests that don't help rankings, each with a suggested fix.
- Response Times — how fast your server answered crawlers.
- Bot Behavior — Crawl Activity by Hour and Verified vs Spoofed Bot Traffic.
- Logs — the raw crawl log with filters by bot, status code, date and URL, and a CSV export (10,000 rows by default, 25,000 at most).
Page Crawl History
Click a URL in Most Crawled URLs, Days Since Crawl, the recently updated list or a raw log row to open Page Crawl History. It shows that page's crawl timeline for the last 24 hours, 7 days, 30 days, 90 days or 12 months, with every bot as its own line, the last and previous crawl times, per-bot totals, response and error summaries and the most recent retained visits. A quiet page is shown as "no recent activity in the retained log", not as proof that a crawler has stopped for good.
Reports & Snapshots
Save a snapshot of your current crawl health with Save snapshot now. Manual snapshots stay until you delete them, so you can compare periods or attach them to client reports; weekly snapshots can be switched on to keep a rolling history.
Real vs. spoofed bots
Some traffic claims to be Googlebot but isn't — scrapers, malicious crawlers, security probes. Linkilo verifies bot identity by reverse DNS plus a forward-confirm check on the source IP, as well as user-agent matching. The Verified vs Spoofed Bot Traffic table (Bot Behavior tab) shows verified and spoofed IPs and the share of hits that came from verified sources.
Verified IPs are remembered for 30 days, so checks are cheap.
GSC Inspect
URLs in the report have a GSC Inspect button. Click it to send the URL through Google Search Console's URL Inspection API. You get back the current indexing state ("Submitted and indexed," "Crawled — currently not indexed," "Excluded by 'noindex'," "Blocked by robots.txt," etc.) without leaving the page.
Results are cached for 7 days, so repeat clicks are instant. Only URLs that belong to your selected Search Console property can be inspected, and Linkilo tells you specifically when Google refuses — expired connection, API not enabled, no access to the property, or quota used up.
Requires Google Search Console connected — see Google Search Console.
Common Questions
Do I need access to my server's access logs?
No. Linkilo records crawler visits as WordPress serves the pages, so it works on shared hosting without log-file access. It only sees requests that reach WordPress — bots stopped at a CDN or firewall before that point are not recorded.
Why is GSC Inspect telling me I'm not connected?
You need to connect Google Search Console first — see Google Search Console. If Google asked you to reconnect, the Index Coverage and Crawler Analyzer reports both say so.
"Crawl ratio" chip explained
The chip compares each URL's crawl count with your other busy pages in the selected window:
- Hot — crawled far more than the baseline; Google really cares about this URL
- Warm — normal attention
- Cool — barely visited
It's a quick way to spot which pages Google values most versus which it barely crawls.
Why are some Googlebot visits marked as spoofed?
That's normal — anyone can set their user-agent to "Googlebot". Linkilo checks the IP with a reverse-DNS lookup and confirms the name resolves back to the same IP. If it doesn't, the visit is flagged as spoofed.
Spoofed traffic often comes from scrapers, security scanners, or people testing for vulnerabilities. Useful to know.
What AI bots does Linkilo track?
Major ones, each as its own line:
- GPTBot (OpenAI training), OAI-SearchBot (ChatGPT search) and ChatGPT-User (user-fetched)
- ClaudeBot (Anthropic), Anthropic-AI and Claude-User
- PerplexityBot and Perplexity-User
- Google-Extended (Gemini training)
- Grok (xAI)
- CCBot (Common Crawl, used by many AI services)
- Plus Meta-ExternalAgent, Bytespider, Amazonbot, Applebot-Extended, Cohere-AI, DuckAssistBot, MistralAI-User and others
SEO tools (AhrefsBot, SemrushBot, Screaming Frog, SEOkicks, SISTRIX, SerpstatBot and more) and social-preview bots are tracked in their own categories. Untick any you don't care about in Card 2. See the Web crawler and bot reference guide for the full list.
What's the retention setting for?
How long Linkilo keeps each crawl event in the database. Old events are pruned automatically in small batches.
- 7 days — minimal disk usage, only useful for "what crawled my site today"
- 30 / 90 days — good for short-term analysis
- 200 days (recommended) — needed for the Days Since Crawl view to find pages that fell off Google's radar
Last reviewed for Linkilo 3.0.0 (October 2026).