Crawl log analysis
Find out whether ChatGPT has actually read your site.
You publish a post and hope Google finds it. You wonder whether ChatGPT has ever read your site. Linkilo logs every bot visit as it happens, tells you which ones are real, and turns it into plain answers inside WordPress. No server access, no log files, no desktop app.
Included in every plan. 30-day money-back guarantee.
The job, as it stands
Why nobody runs a log file analyzer for WordPress twice
Get SSH access. Find the access log. Download a few hundred megabytes of it. Load it into something that can read it. Work out which user agents are real crawlers. Build the pivot. Do it again next month.
By hand, that means
- Get SSH access, or ask your host for the raw access logs.
- Download them. On a busy site that is hundreds of megabytes a month.
- Load them into something that can parse them, and filter to crawler user agents.
- Check which of those are real crawlers and which are spoofing Googlebot.
- Map every hit to a page and count visits, status codes and last-seen dates.
- Compare that with the pages you care about. Then do all of it again next month.
Linkilo does every step of that.
What it costs to leave it
Until you look, you are guessing which pages Google actually visits. The usual guess is wrong in the expensive direction: crawl budget spent on redirects and paginated archives while the guide you spent a week on has not been fetched since spring.
Somebody has to do that: you, or somebody you pay.
What you get back
The whole exercise, and the reason you have never actually done it.
What it looks like instead
You open a page in WordPress and see which of your pages Google crawled this month, and which ones GPTBot, ClaudeBot and PerplexityBot did.
How soon
Reads what your server already writes. Nothing to download.
Who does the work
Deterministic
Straight parsing of your access log against a maintained list of crawler user agents. Nothing is inferred and nothing is sampled.
Time figures describe the shape of the job (an afternoon, a day, a week), not a promise about your site. Your archive is your archive.
You are making SEO decisions without seeing the full picture.
Search Console tells you about pages Google chose to report on, days after the fact. Your logs tell you what Google actually did, right now. Those are not the same thing, and only one of them is the truth.
What you are working with today
- Search Console is always behind, so a problem hurts you for days before you see it
- Most WordPress hosts will not give you raw server logs at all
- Even with the logs, reading them properly is somebody’s full-time job
- AI crawlers visit daily and your analytics ignores them completely
- Some of your best articles have not been crawled in months and you would never know
What you get instead
- Every bot visit logged in real time, inside WordPress
- A ranked list of pages going stale, before they slip out of the index
- GPTBot, ClaudeBot, PerplexityBot and 40 more, tracked by name
- Verified crawlers separated from anything pretending to be one
- A polished PDF or CSV for the client, in about thirty seconds
Built around the questions you actually ask
Every one of these has a screen, and most of them have a one-click action attached.
“Is Google finding my new content?”
Every Googlebot visit is logged in real time, so you can confirm a new post was fetched within hours of publishing, or catch it immediately when it was not.
“Which pages are at risk?”
A ranked list of pages not crawled lately, plus a separate list of pages you have edited that Google has not returned to since.
“Is my site healthy overall?”
One 0–100 score blending crawl frequency, page coverage, error rate and response speed. Each component breaks down the real issues and what to do about them.
“Where is my crawl budget going?”
See exactly which URL types are eating bot time without earning search traffic, then copy a ready-made robots.txt block to stop the waste.
“What does Google think of this page?”
Click Inspect on any URL to see Google’s own verdict: indexed, not indexed, blocked, or canonicalised elsewhere. Without leaving WordPress.
“Is this crawler traffic even real?”
Each major bot is verified against its published source IP ranges, giving you a clean verified-versus-spoofed split. Impostors never pollute your numbers.
“Are AI tools reading my content?”
See which AI crawlers reach you, from GPTBot and ClaudeBot to PerplexityBot and 40 more, and whether they are finding your most important pages or missing them.
“Does my sitemap match reality?”
Two lists side by side: pages in your sitemap Google never crawls, and pages Google does crawl that are not in your sitemap. Quick wins on both sides.
“Will I know when something changes?”
Email alerts fire when Googlebot goes quiet, errors spike, or a bot starts hammering you, with automatic anomaly detection on top.
“Can I prove this to a client?”
Save a snapshot, compare any two periods with change arrows, and export a polished PDF or CSV that is ready to send without editing.
“Will it slow my site down?”
A bot visit costs a single lightweight database insert, under 0.01s. Nothing runs for human visitors, and the analysis happens in the background.
“Will it fill up my database?”
Retention is configurable and old rows are pruned automatically, so the table stays bounded rather than growing forever on a busy site.
Bots have limited time for your site. Make them spend it well.
Crawl waste is bot time spent on URLs that earn you nothing. If a quarter of every Googlebot visit lands on pages that do not exist, do not matter, or are just parameter noise, that is a quarter of your discovery budget gone.
See which URL types are eating the budget
Tag archives, paginated listings, faceted navigation, parameter URLs, feeds. Individually harmless; collectively they can absorb a third of everything a crawler does on your site while earning no search traffic at all.
Linkilo groups crawl activity by URL pattern and puts it next to the search traffic each pattern actually earns. The imbalance is usually obvious within about ten seconds of looking at it, and you get a ready-made robots.txt block to copy rather than a research project.
- Crawl volume by content type and URL pattern
- Status codes bots hit, including 404s and redirect chains
- A ready-made robots.txt block for the patterns worth excluding
- Pages that are crawled often but earn nothing, flagged
Posts and pages 48%
Tag archives 27%
Paginated listings 14%
404s and redirects 11%
52% of Googlebot’s time is going to URLs earning no search traffic.
Anyone can claim to be Googlebot. We check.
Setting a user agent string to “Googlebot” takes one line of code, and plenty of scrapers do exactly that. If your crawl stats count those, every conclusion you draw from them is wrong in a direction you cannot see.
Linkilo verifies each major crawler against its published source IP ranges and shows you a clean verified-versus-spoofed split. Real Google, real Bing, real OpenAI, real Anthropic, and separately, the impostors.
- Source-IP verification against published crawler ranges
- Verified and spoofed shown as separate figures, never merged
- Verification results cached for 30 days, so it costs nothing to maintain
- Aggressive bots surfaced so you can rate-limit them
Claiming to be Googlebot
12,481 verified 1,702 spoofed
Spoofed hits are excluded from every chart on this page.
Skip the Search Console tab-switching
Click Inspect on any URL and you see what Google itself says about it: is it indexed, is it indexable, is robots.txt blocking it, has Google chosen a different canonical than the one you specified.
That last one catches a surprising number of problems. A page can be crawled, healthy, and completely absent from search because Google decided some other URL was the canonical version, and nothing in your own reporting would ever tell you.
- Index status, indexability and robots.txt blocking per URL
- Google’s chosen canonical shown against your declared one
- Runs against your connected Search Console property
- Sitemap-versus-reality comparison in both directions
/guides/travel-humidors/
Indexed No
Indexable Yes
robots.txt Allowed
Google canonical /guides/humidors/
Crawled and healthy, but Google picked a different canonical.
Are AI tools quoting your site?
When somebody asks ChatGPT, Claude, Perplexity or Gemini a question, those tools go and read web pages to answer it. If they cannot reach yours, you are invisible in a channel that is growing every month, and no traditional SEO tool will tell you.
Forty-plus crawlers, tracked by name
Being cited in an AI answer starts with that crawler having read the page. Linkilo tracks GPTBot, ClaudeBot, PerplexityBot, Google-Extended and dozens more individually, alongside the traditional search bots.
It also separates why they came. A crawl for training data is a different signal from a live fetch made because a real person asked a question this second, and only one of those means you are actively being used to answer someone.
- 40+ bots identified individually, not lumped into “other”
- Live answer fetches separated from training-data crawls
- Per-page AI crawl history, so you can see which content they favour
- AI coverage compared directly against search-engine coverage
Live answers 156
Training data 699
A real user asked 38
Pages seen by at least one AI crawler
Client reports that take thirty seconds, not thirty minutes
Every month you have to show somebody that the work is paying off. Save a snapshot today, save another one next month, and Linkilo lays them side by side with change arrows on every metric.
That is a much better conversation than a screenshot of Search Console. “Googlebot now crawls 91% of your pages, up from 64%, and these eleven pages went from never-crawled to indexed” is a sentence that renews a retainer.
- Unlimited saved snapshots
- Compare any two periods with change indicators
- Export a polished PDF or CSV, ready to send
- Email alerts on anomalies, so you hear about it before the client does
Crawl frequency 92 / 100
Page coverage 71 / 100
Error rate 88 / 100
Response speed 85 / 100
Coverage is the weak component: 41 pages have never been crawled.
The crawl report, exactly as it ships.
Which bots hit the site, how often, what they wasted the budget on, and which URLs answered slowly. All of it read from your real access log rather than a simulated crawl.
Crawler Analyzer Analyze how search engine bots and AI crawlers interact with your site. Track crawl activity, coverage, errors, and performance.
Crawl Activity
Status Codes
Most Crawled URLs Top 20 URLs by raw hit count in the date window and bot you have selected at the top of the dashboard. The percentage chip on each row compares this URL to the mean crawl count of the busiest 25% of URLs on your site, so >100% means "more crawled than your typical busy page". A "crawl trap" badge appears when at least 10% of crawls to this URL returned a 4xx or 5xx — those URLs are wasting crawl budget and deserve a redirect, fix, or robots.txt rule.
| URL | Crawls | Avg Response | Actions |
|---|---|---|---|
| yoursite.com/best-cigar-humidors-2026/ | 1,942 | 212ms | |
| yoursite.com/cigar-humidity-chart-69-vs-72/ | 1,618 | 198ms | |
| yoursite.com/old-category/humidors/ | 1,230 crawl trap | 1204ms | |
| yoursite.com/how-to-season-a-new-humidor/ | 1,104 | 244ms | |
| yoursite.com/?s=humidor&orderby=price | 866 mixed | 980ms | |
| yoursite.com/travel-humidor-buying-guide/ | 702 | 231ms | |
| yoursite.com/wp-json/wp/v2/posts?page=14 | 540 | 388ms |
Score Breakdown How your health score is calculated across four key components.
Each factor contributes 25% to your overall score:
Issues Found Specific problems detected during analysis that are affecting your crawl health.
Recommendations to Improve Your Score Actionable steps to fix issues and boost your crawler health score.
- Redirect /old-category/humidors/ to /cigar-humidors/ (or return 410) to stop 1,104 wasted crawls a week.
- Add internal links to the 38 uncrawled pages from your pillar posts (Best Cigar Humidors of 2026, How to Season a New Humidor).
- Block parameter URLs such as ?orderby= and ?s= in robots.txt to free crawl budget for content pages.
- Enable page caching for bot traffic to bring GPTBot and PerplexityBot response times under 500ms.
Understanding the Score Components Detailed explanation of each health score factor.
Crawl Frequency (25%) How often bots visit your site. Regular crawling indicates search engines find your content valuable.
How often Googlebot visits your site. Daily visits = 100%. Less frequent visits lower your score.
Coverage (25%) What percentage of your published content has been crawled. Higher coverage means better indexing.
Percentage of your pages that have been crawled in the last 30 days. Higher coverage = better indexing potential.
Error Rate (25%) The percentage of crawl requests returning errors. Lower is better — aim for under 5%.
Percentage of crawl requests returning 4xx/5xx errors. Lower error rate = healthier site.
Response Time (25%) How fast your server responds to crawler requests. Faster responses mean more pages crawled per visit.
Average server response time. Under 500ms = excellent. Over 1 second hurts crawl efficiency.
Crawler Coverage Percentage of your published content visited by each crawler.
| Bot | Unique URLs Crawled | Coverage % | Coverage Visualization |
|---|---|---|---|
| Googlebot | 274 | 87.8% | |
| Bingbot | 231 | 74.0% | |
| GPTBot | 188 | 60.3% | |
| ClaudeBot | 142 | 45.5% | |
| PerplexityBot | 97 | 31.1% | |
| OAI-SearchBot | 76 | 24.4% |
Recommendations to Improve Crawler Coverage:
- Optimize your site's loading speed.
- Ensure your XML sitemap is up-to-date and submitted to search engines.
- Regularly check for and fix broken internal and external links.
- Improve internal linking to ensure all important pages are easily discoverable.
- Ensure your robots.txt file is not blocking important crawlers or content.
Recently Updated Pages Needing Re-Crawl Pages edited in the last 60 days where the selected bot either has never crawled them, or the most recent crawl predates your latest edit. These are the pages most likely to show stale content in search/AI results.
5 of 41 page(s) modified in the last 60 days need Googlebot attention: 2 never crawled, 3 crawled before the latest edit.
| Title | Type | Last Modified | Last Crawl | Status & Actions |
|---|---|---|---|---|
| Travel Humidor Buying Guide | Post | 2026-09-28 | Never | never crawled Edit View |
| How to Season a New Humidor | Post | 2026-09-30 | 2026-09-12 | stale · last crawl 23d ago Edit View |
| Cigar Humidity Chart (69% vs 72%) | Post | 2026-10-01 | 2026-09-24 | stale · last crawl 11d ago Edit View |
| Boveda vs Humidity Beads: Which Lasts Longer? | Post | 2026-09-19 | Never | never crawled Edit View |
| Humidor Seasoning FAQ | Page | 2026-09-26 | 2026-09-20 | stale · last crawl 15d ago Edit View |
Pages flagged "never crawled" have no log entry for the selected bot at all. Pages flagged "stale" were crawled, but the most recent crawl predates your last edit. For either case, consider submitting the URL via Google Search Console / Bing Webmaster Tools, or improving internal links to it.
Understanding Days Since Last Crawl Guide to crawl recency and deindex risk levels.
The longer a page goes without being crawled (or re-crawled) by a search engine bot like Googlebot, the higher the potential risk that it might be considered stale or less important, which could eventually affect its indexing status. This report helps identify such pages.
General Guideline (varies by site & bot):
- Low Risk (< 100 days): Page is being crawled regularly. No action needed.
- Medium Risk (100-130 days): Monitor closely, consider updating content or improving internal links.
- High Risk (131-150 days): Higher attention needed. Update content and request re-indexing.
- Critical Risk (151+ days or Never): These might be at greater risk of deindexing if they are important pages.
Content Status by Last Crawl Date Range Distribution chart showing how many pages fall into each crawl recency bucket.
Pages by Days Since Last Crawl (Selected Bot & Risk Level) Detailed table of pages with their last crawl date and risk level.
| Title | Type | Last Crawl | Days Since | Risk Level | Actions |
|---|---|---|---|---|---|
| Spanish Cedar Tray Humidor Review | post | 2026-04-02 | 186 | Critical | Edit |
| Humidor Hygrometer Calibration (Salt Test) | post | 2026-05-01 | 157 | Critical | Edit |
| Cigar Storage Without a Humidor | post | 2026-05-18 | 140 | High | Edit |
| Acrylic vs Wood Humidors | post | 2026-06-12 | 115 | Medium | Edit |
| Our Story | page | Never | Never | Critical | Edit |
| Best Cigar Humidors of 2026 | post | 2026-10-05 | 0 | Low | Edit |
Google Search Console Insights Aggregate report from URLs you have inspected via Google Search Console. Shows indexing verdict mix, coverage states, canonical mismatches, robots.txt blocks, and pages where Google's last crawl predates your last edit. Use Bulk Inspect to grow the dataset — Linkilo throttles requests to stay under Google's 600/min limit.
Which of your pages Google has actually indexed — verdicts, coverage states, canonical mismatches and re-inspection — now lives in the dedicated Index Coverage report. That keeps this analyzer focused on crawl behavior (which bots hit what, and how recently). So far 286 URLs inspected, 241 indexed.
Open Index Coverage →Crawl Waste Analysis Crawls that did not help SEO/AI visibility. Trend arrows compare this window to the previous equal-length window. The "Fix top 3" estimate sums the hits in your three biggest fixable categories. Results are cached for 10 minutes — click Refresh to force a recompute. Computed in 1.4 s · cached for 10 minutes — refresh to recompute.
Where bots spent your crawl budget How Googlebot distributed its crawls across content categories on your site. Big slices outside "Content Pages" usually mean crawl budget is being eaten by archives, parameter URLs, errors, or media.
Efficiency 77% — 19,842 crawls, 4,566 wastedDaily waste rate (last 30 days) Percentage of each day's bot crawls that fell into a waste category. Watch for sustained climbs — they usually mean a parameter explosion or a new redirect chain has started.
Top 3 fixes (ranked by impact)
Update internal links to point at the final URL. 2 chains are three hops deep, starting at /old-category/humidors/.
Redirect /old-category/humidors/ to /cigar-humidors/ or return 410 so bots stop retrying it.
Block sort and search parameters in robots.txt. They produce the same content as the canonical page.
Rules already present in your existing robots.txt have been excluded. Review carefully before adding — overly aggressive rules can hide real content from Google.
User-agent: * Disallow: /*?orderby= Disallow: /*?s= Disallow: /wp-json/
Full breakdown
Hard waste — fix urgently
Bots followed a redirect instead of landing on the page. Each hop is a wasted request.
Suggested fix: Update the internal links that still point at the old URL.
- https://yoursite.com/old-category/humidors/ (684 hits)
- https://yoursite.com/shop/boveda-84/ (212 hits)
Requests that returned 404/410. Bots keep retrying these for weeks.
Suggested fix: Redirect them to the closest live page or return 410 Gone.
- https://yoursite.com/old-category/humidors/ (1104 hits)
Server errors. Googlebot slows its crawl rate when it sees these.
Suggested fix: Check the PHP error log for the timestamps in the Logs tab.
Probable waste — usually worth fixing
Sort, filter and search parameters that duplicate a canonical page.
Suggested fix: Disallow the parameter patterns in robots.txt and set canonicals.
- https://yoursite.com/?s=humidor&orderby=price (866 hits)
Archive pages with little unique content.
Suggested fix: Noindex archives you do not need in search, or trim pagination depth.
- https://yoursite.com/2024/03/ (118 hits)
- https://yoursite.com/author/editor/page/9/ (64 hits)
wp-json and RSS endpoints fetched by crawlers.
Suggested fix: Disallow /wp-json/ for crawlers that do not need it.
- https://yoursite.com/wp-json/wp/v2/posts?page=14 (540 hits)
Configuration waste — site setting
The same page fetched under two spellings of its URL.
Suggested fix: Enforce one canonical form with a single 301 at the server level.
- https://yoursite.com/Best-Cigar-Humidors-2026 (212 hits)
Sitemap Health (Sitemap vs. Crawled) Pages listed in your sitemap that Googlebot has not crawled in the last 30 days (often: thin content, low link equity, or sitemap noise). Plus URLs Googlebot crawled successfully that are NOT in your sitemap (orphan content that should be added).
In sitemap but uncrawled (top 25)
Pages your sitemap claims matter, that Googlebot has not actually fetched in 30 days. Often: thin content, deep pagination, or low link equity.
- https://yoursite.com/humidor-glossary/
- https://yoursite.com/cigar-cutters-guide/
- https://yoursite.com/about/press/
Crawled but missing from sitemap (top 25)
Real-looking content URLs Googlebot is finding via internal links, but your sitemap does not list. Add the legitimate ones to the sitemap so they get prioritized.
- https://yoursite.com/humidor-seasoning-faq/
- https://yoursite.com/tag/boveda/
Response Time Analytics Server response times broken down by bot and URL. Identify slow pages that waste crawl budget.
By Bot
| Bot | Avg | Min | Max | Requests |
|---|---|---|---|---|
| Googlebot | 262ms | 48ms | 1840ms | 9,411 |
| Bingbot | 301ms | 52ms | 2210ms | 3,122 |
| GPTBot | 612ms | 90ms | 4120ms | 2,804 |
| ClaudeBot | 388ms | 71ms | 2980ms | 1,642 |
| PerplexityBot | 344ms | 66ms | 2450ms | 1,311 |
Trend
Slowest Pages
| URL | Avg Time | Hits |
|---|---|---|
| yoursite.com/old-category/humidors/ | 1204ms | 1230 |
| yoursite.com/?s=humidor&orderby=price | 980ms | 866 |
| yoursite.com/best-cigar-humidors-2026/comment-page-3/ | 844ms | 212 |
| yoursite.com/wp-json/wp/v2/posts?page=14 | 388ms | 540 |
Bot Behavior Analytics Patterns in how each crawler navigates your site — frequency, preferred paths, and activity timing.
Crawl Activity by Hour
Content Type Distribution
Active Bots (Last 7 Days)
Crawl Insights (LLM Era) Modern crawl signals that matter for AI search visibility: how AI bots are split between training/search/user-fetched, mobile vs desktop Googlebot, 304-Not-Modified efficiency, and any bot showing an unusual spike or drop today.
AI bot traffic by purpose (last 7 days)
Googlebot: mobile vs desktop
Mobile-first crawling, as expected.
304 Not Modified efficiency
Healthy use of conditional requests — your server saves work on unchanged pages.
Today's anomalies (vs 7-day average)
- 📈 GPTBot — today 612 hits vs 7-day avg 400 (1.5×)
Verified vs Spoofed Bot Traffic Anyone can put "Googlebot" in their User-Agent. Linkilo verifies each source IP via reverse DNS + forward-confirm — exactly the procedure Google, Bing, OpenAI and Anthropic publish for their bots. "Spoofed" rows are someone pretending to be that bot. Results are cached for 30 days; the first load is slow as new IPs are checked.
99% of crawl hits in the last 7 days came from verified sources. 142 hits were from spoofers impersonating real bots.
| Bot | Verified IPs | Spoofed IPs | Unknown | Verified Hits | Sample Spoofers |
|---|---|---|---|---|---|
| Googlebot 9411 hits / 41 IPs | 38 | 3 | 0 | 99% | 203.0.113.17, 198.51.100.9 |
| Bingbot 3122 hits / 22 IPs | 21 | 1 | 0 | 98% | 192.0.2.44 |
| GPTBot 2804 hits / 17 IPs | 17 | 0 | 0 | 100% | — |
| ClaudeBot 1642 hits / 9 IPs | 9 | 0 | 0 | 100% | — |
Crawl Logs Raw server log entries showing every crawl request with full details.
| Time | Bot | URL | Status | Response |
|---|---|---|---|---|
| Oct 5, 09:42 AM | googlebot | /best-cigar-humidors-2026/ | 200 | 214ms |
| Oct 5, 09:41 AM | gptbot | /cigar-humidity-chart-69-vs-72/ | 200 | 598ms |
| Oct 5, 09:41 AM | googlebot | /old-category/humidors/ | 404 | 1188ms |
| Oct 5, 09:39 AM | bingbot | /shop/boveda-84/ | 301 | 96ms |
| Oct 5, 09:38 AM | perplexitybot | /how-to-season-a-new-humidor/ | 200 | 338ms |
| Oct 5, 09:36 AM | claudebot | /travel-humidor-buying-guide/ | 200 | 402ms |
| Oct 5, 09:35 AM | googlebot | /wp-json/wp/v2/posts?page=14 | 200 | 384ms |
| Oct 5, 09:33 AM | oai-searchbot | /humidor-seasoning-faq/ | 304 | 61ms |
| Oct 5, 09:31 AM | googlebot | /?s=humidor&orderby=price | 500 | 2140ms |
Save a snapshot Captures total crawls, unique URLs, status-code distribution, bot mix, top crawled URLs, coverage by bot, health score, crawl waste, and GSC summary at this moment in time.
Saved snapshots Manual snapshots stay until you delete them. Automated weekly snapshots keep a rolling history. Use View to open the printable report, Compare to diff against another snapshot, CSV to export raw numbers, or Delete to remove an obsolete snapshot.
Quick: open a live report (no save)
Generate a printable report from the current numbers without saving a snapshot. Useful for ad-hoc client emails.
Download 7-day CSVAlert Configuration Set up automated notifications when crawl metrics exceed your thresholds.
| Alert Name | Type | Conditions | Recipients | Status | Actions |
|---|---|---|---|---|---|
| Googlebot Inactivity Alert | No Crawl Activity | No Googlebot crawl for 24 hours | Admin | Active | |
| Error spike | High Error Rate | Error rate > 5% | Admin + Custom | Active | |
| GPTBot slow responses | Slow Response | Response time > 1000ms | Admin | Inactive |
What Add Alert opens
Understanding Web Crawlers & AI Bots A reference guide to the different types of bots that visit your site and how to manage them.
Web crawlers are automated programs that visit websites to index content, gather data, or generate previews. Understanding which bots visit your site helps you optimize for search engines, manage AI training opt-outs, and ensure proper social media sharing.
Search Engine Crawlers Crawlers from major search engines that index your content for search results.
These bots index your content for search results. Blocking them removes your site from their search.
| Bot | Company | Purpose | Recommendation |
|---|---|---|---|
| Googlebot | Primary crawler for Google Search. Essential for visibility. | Always Allow | |
| Bingbot | Microsoft | Indexes for Bing Search and powers Copilot features. | Always Allow |
| DuckDuckBot | DuckDuckGo | Privacy-focused search engine with growing user base. | Recommended |
| YandexBot | Yandex | Leading search engine in Russia and CIS countries. | Regional |
| Baiduspider | Baidu | China's largest search engine. | Regional |
| Applebot | Apple | Powers Siri and Spotlight search suggestions. | Recommended |
AI/LLM Training Crawlers Bots that collect content for AI model training. You can safely block these without affecting search rankings.
These bots collect content for AI training. You can block them via robots.txt without affecting search rankings.
Tip: Blocking these will NOT affect your Google/Bing search rankings.
| Bot | Company | Purpose | Traffic Level |
|---|---|---|---|
| GPTBot | OpenAI | Collects data for GPT models. Highest volume AI crawler (~30% of AI traffic). | Very High |
| Google-Extended | For Gemini AI training. Separate from Googlebot - safe to block. | High | |
| ClaudeBot | Anthropic | Trains Claude AI models. Respects robots.txt. | Medium |
| Meta-ExternalAgent | Meta | For Meta's Llama AI models (~19% of AI traffic). | Very High |
| PerplexityBot | Perplexity | Fastest growing AI crawler (157,000%+ growth in 2024). | High |
| CCBot | Common Crawl | Open datasets used by many AI companies. | Medium |
| Bytespider | ByteDance | TikTok parent company's AI crawler. | Declining |
| Grok | xAI | Trains Grok AI for X (Twitter). | Low-Medium |
AI Assistant Crawlers (User-Triggered) Bots that fetch your content when AI users request information. These can generate referral traffic.
These fetch pages when users ask AI assistants questions. They can drive referral traffic when AI cites your content.
Recommendation: Consider ALLOWING these - they represent real users and can send traffic to your site.
| Bot | Company | Purpose | Referral Potential |
|---|---|---|---|
| ChatGPT-User | OpenAI | Fetches pages when ChatGPT users ask questions (2,825%+ growth). | High |
| Claude-User | Anthropic | Fetches pages when Claude users request web content. | Medium |
| Perplexity-User | Perplexity | Fetches for search answers. Note: May sometimes ignore robots.txt. | High |
| OAI-SearchBot | OpenAI | OpenAI's dedicated search crawler for ChatGPT search. | Medium-High |
Social Media Crawlers Bots from social platforms that generate link previews when your content is shared.
Generate link previews when URLs are shared. Blocking them breaks previews.
| Bot | Platform | Purpose | Recommendation |
|---|---|---|---|
| facebookexternalhit | Meta | Link previews for Facebook and Instagram. | Always Allow |
| Twitterbot | X (Twitter) | Twitter/X card previews. | Always Allow |
| LinkedInBot | Link previews for LinkedIn posts. | Always Allow | |
| Slackbot | Slack | Link previews in Slack workspaces. | Recommended |
| Discordbot | Discord | Link previews in Discord. | Recommended |
SEO Tool Crawlers Crawlers from SEO tools that analyze your site. Can be resource-intensive.
Gather data for SEO analysis. Can be resource-intensive. Consider rate-limiting if performance issues occur.
| Bot | Tool | Purpose | Activity Level |
|---|---|---|---|
| AhrefsBot | Ahrefs | Backlink analysis and site auditing. | Very High |
| SemrushBot | Semrush | Keyword and competitor research. | High |
| MJ12bot | Majestic | Link intelligence database. | Medium |
| DotBot | Moz | Domain authority calculations. | Low |
Quick robots.txt Reference Common robots.txt rules for allowing or blocking specific crawlers.
Your robots.txt: https://yoursite.com/robots.txt
Block AI Training
User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: meta-externalagent Disallow: /
Allow AI Assistants
User-agent: ChatGPT-User Allow: / User-agent: Claude-User Allow: / User-agent: Perplexity-User Allow: / User-agent: OAI-SearchBot Allow: /
Note: Not all bots respect robots.txt. For stronger control, use .htaccess rules or a WAF.
Recreated from core/Linkilo/serverloganalyzer. Same metrics, same buckets, same next move.
What is actually running behind those screens
Built for anyone who cares about search traffic
If your audience finds you through Google or through an AI tool, you need to know what those tools are actually seeing.
SEO professionals
Search Console’s delay means you react days late, every time.
With Linkilo: decisions get made on data that is minutes old, so issues are caught before they cost rankings.
Content publishers
You never find out whether the latest post was even discovered.
With Linkilo: confirm a fetch within hours, and see which older articles need a refresh because nobody is coming back.
Ecommerce
Product pages get lost in faceted-navigation noise.
With Linkilo: see where the budget is going and catch 404s from discontinued products before they hurt the store.
Marketing agencies
Proving technical SEO value is a monthly slog.
With Linkilo: comparable period-over-period numbers and a one-click PDF that justifies the retainer.
Larger sites
With thousands of pages, important ones slip through unnoticed.
With Linkilo: the patterns become obvious in a single dashboard load instead of a spreadsheet exercise.
AI-aware publishers
Your audience increasingly asks ChatGPT instead of Googling.
With Linkilo: you can finally see whether those tools reach your content, and which pages they favour.
Three ways to learn what bots do on your site. Two are painful.
Raw logs are complete but unreadable. Search Console is readable but delayed and partial. This is the third option.
| What you want to do | Linkilo | Raw server logs | Search Console |
|---|---|---|---|
| See bot activity in real time | Yes | Yes | No |
| Without server or host access | Yes | No | Yes |
| Without learning a log format | Yes | No | Yes |
| AI crawlers identified by name | Yes | If you parse them | No |
| Verified vs spoofed bot split | Yes | Manual | No |
| Crawl waste analysis with a fix | Yes | Manual | No |
| Never-crawled page list | Yes | Manual | Partial |
| Cross-referenced with your internal links | Yes | No | No |
| One-click client PDF | Yes | No | No |
| Period-over-period comparison | Yes | Manual | Limited |
| Works on non-WordPress sites | No | Yes | Yes |
| Sees requests served entirely from CDN cache | No | Yes | No |
Two honest limitations above. If you need enterprise-scale log analysis across many stacks, see the Screaming Frog comparison, which explains exactly when that is the better tool.
From install to insight in minutes
No server configuration. No log files to download. No technical expertise required.
Install and activate
One-click WordPress install. Works with any host, including shared hosting where you will never see a server log.
It starts watching immediately
Every bot visit is logged automatically from that moment. Optionally connect Search Console to open up URL inspection.
Insights appear within a day or two
Once there is enough activity you get meaningful patterns, a health score, and a list of things worth fixing.
Act, alert and report
Apply the suggested fixes, set anomaly alerts, save snapshots, and export the report when somebody asks what changed.
The questions that decide it
Including the one about database size, which is the right thing to worry about.
No. A bot visit costs one lightweight database insert, measured under 0.01s. There is no processing on the request and nothing at all runs for human visitors. Aggregation and analysis happen in the background on WP-Cron.
Retention is configurable and old rows are pruned automatically, so the table stays bounded rather than growing forever. The default keeps enough history for month-over-month comparison, and saved snapshots preserve the summary figures long after the raw rows are gone.
Screaming Frog analyzes log files you have already obtained from your server, and it is more powerful at that job. Linkilo does the logging itself inside WordPress, so there is no export step, no server access, and the data sits next to your link data. The full comparison, including where Screaming Frog wins, has its own page.
Bots are identified by user agent and then verified against the published source IP ranges of the major crawlers. Anything claiming to be Googlebot from an address Google does not own is flagged as spoofed and excluded from your statistics rather than silently counted.
This is the honest limitation of in-application logging: a request served entirely by a CDN or full-page cache without ever reaching PHP cannot be recorded by a WordPress plugin. Raw server or CDN logs will always be the more complete record. For most WordPress sites the gap is small, but you should know it exists.
Only for URL inspection and the sitemap comparison. Crawl logging, bot verification, waste analysis, health scoring and the AI crawler tracking all work with nothing connected.
Yes, and that is largely the point. Shared hosts are exactly where you cannot get raw logs, so this is often the only way to see crawl data at all on those sites.
PDF for the polished client-facing version, CSV when you want to do your own analysis. Snapshots let you compare any two periods with change indicators.
Orphan Pages Tool & Reporting
Find the pages nothing links to, the ones Google struggles to find, and get them connected.
Broken Link Checker & One-Click Repair
Find every dead link on your site and fix it in one click. No spreadsheets, no manual editing.
Redirection Manager
Catch 404s as they happen and redirect them before anyone notices the traffic went missing.
AI Bot & Robots.txt Checker (free)
Check which AI crawlers your robots.txt actually lets in, and which ones it blocks.
Stop guessing. Start seeing.
Find what is being missed, fix what is being wasted, and prove the work to whoever is asking. No server access, no log files, no desktop app.
30-day money-back guarantee. Every feature is included in every plan.