Honest comparison
Linkilo vs Screaming Frog Log File Analyser: crawl data without the log files.
The Log File Analyser is the standard desktop tool for reading server logs, and if you have raw logs and a workstation it is excellent. Linkilo records crawler visits inside WordPress as they happen, with no log files and no desktop app. Here is what each one sees, what each one misses, and which one fits a site you run rather than a site you audit.
If you only read one paragraph
These are different shapes of the same job. Screaming Frog reads logs you export from a server, on your desktop, for any platform, with unmatched depth: it will chew through millions of lines, verify bots several ways and compare a crawl against a log to find orphans. The cost is the ritual: get the raw logs, which many managed WordPress hosts will not hand over, import them monthly, and act on what you find somewhere else. Linkilo lives in the site. Every bot visit is logged as it happens, verified, kept for as long as you choose, tied to the post it hit, and turned into a stale-pages list, crawl-waste groups, a health score and email alerts, with the link-repair tools one click away. It cannot see hits your cache or CDN served before WordPress ran. If you audit other people’s sites from logs, buy Screaming Frog. If you run WordPress sites and want to know what Google and the AI crawlers did this morning, that is Linkilo.
Four things that frame the comparison
Including the one where Screaming Frog is plainly ahead.
Where it runs
- Linkilo
- Inside WordPress, continuously
- Screaming Frog Log File Analyser
- A Java desktop app on Windows, macOS or Linux, run when you import logs
Its system requirements call for a few gigabytes of free memory and disk space of about two and a half times the log size. Linkilo needs the site it is already on.
What it sees
- Linkilo
- Every request that reached WordPress
- Screaming Frog Log File Analyser
- Every request that reached the server, including cached, CDN and static hits
Screaming Frog wins this one. A full-page cache or CDN answers some bot requests before WordPress runs, and Linkilo never sees those. The server log does.
Where the logs come from
- Linkilo
- Nowhere. The site records them itself
- Screaming Frog Log File Analyser
- You download raw access logs from the host, ideally three months of them, and import
Its own guide lists which managed hosts give you logs and how. Several only expose the last few hundred lines or thirty days, and CDN-served hits are often only in the CDN dashboard.
What it costs
- Linkilo
- From $59 a year for one site, every feature included
- Screaming Frog Log File Analyser
- £99 per user per year; free up to 1,000 log lines and one project
Priced per person, not per site, so an agency pays per analyst. Linkilo is priced per site.
The crawl log, exactly as it ships.
Bot visits as they happen, verified, by page and by crawler, with the stale-pages list and the health score built from them. Nothing was exported, imported or uploaded to get here.
Crawler Analyzer Analyze how search engine bots and AI crawlers interact with your site. Track crawl activity, coverage, errors, and performance.
Crawl Activity
Status Codes
Most Crawled URLs Top 20 URLs by raw hit count in the date window and bot you have selected at the top of the dashboard. The percentage chip on each row compares this URL to the mean crawl count of the busiest 25% of URLs on your site, so >100% means "more crawled than your typical busy page". A "crawl trap" badge appears when at least 10% of crawls to this URL returned a 4xx or 5xx — those URLs are wasting crawl budget and deserve a redirect, fix, or robots.txt rule.
| URL | Crawls | Avg Response | Actions |
|---|---|---|---|
| yoursite.com/best-cigar-humidors-2026/ | 1,942 | 212ms | |
| yoursite.com/cigar-humidity-chart-69-vs-72/ | 1,618 | 198ms | |
| yoursite.com/old-category/humidors/ | 1,230 crawl trap | 1204ms | |
| yoursite.com/how-to-season-a-new-humidor/ | 1,104 | 244ms | |
| yoursite.com/?s=humidor&orderby=price | 866 mixed | 980ms | |
| yoursite.com/travel-humidor-buying-guide/ | 702 | 231ms | |
| yoursite.com/wp-json/wp/v2/posts?page=14 | 540 | 388ms |
Score Breakdown How your health score is calculated across four key components.
Each factor contributes 25% to your overall score:
Issues Found Specific problems detected during analysis that are affecting your crawl health.
Recommendations to Improve Your Score Actionable steps to fix issues and boost your crawler health score.
- Redirect /old-category/humidors/ to /cigar-humidors/ (or return 410) to stop 1,104 wasted crawls a week.
- Add internal links to the 38 uncrawled pages from your pillar posts (Best Cigar Humidors of 2026, How to Season a New Humidor).
- Block parameter URLs such as ?orderby= and ?s= in robots.txt to free crawl budget for content pages.
- Enable page caching for bot traffic to bring GPTBot and PerplexityBot response times under 500ms.
Understanding the Score Components Detailed explanation of each health score factor.
Crawl Frequency (25%) How often bots visit your site. Regular crawling indicates search engines find your content valuable.
How often Googlebot visits your site. Daily visits = 100%. Less frequent visits lower your score.
Coverage (25%) What percentage of your published content has been crawled. Higher coverage means better indexing.
Percentage of your pages that have been crawled in the last 30 days. Higher coverage = better indexing potential.
Error Rate (25%) The percentage of crawl requests returning errors. Lower is better — aim for under 5%.
Percentage of crawl requests returning 4xx/5xx errors. Lower error rate = healthier site.
Response Time (25%) How fast your server responds to crawler requests. Faster responses mean more pages crawled per visit.
Average server response time. Under 500ms = excellent. Over 1 second hurts crawl efficiency.
Crawler Coverage Percentage of your published content visited by each crawler.
| Bot | Unique URLs Crawled | Coverage % | Coverage Visualization |
|---|---|---|---|
| Googlebot | 274 | 87.8% | |
| Bingbot | 231 | 74.0% | |
| GPTBot | 188 | 60.3% | |
| ClaudeBot | 142 | 45.5% | |
| PerplexityBot | 97 | 31.1% | |
| OAI-SearchBot | 76 | 24.4% |
Recommendations to Improve Crawler Coverage:
- Optimize your site's loading speed.
- Ensure your XML sitemap is up-to-date and submitted to search engines.
- Regularly check for and fix broken internal and external links.
- Improve internal linking to ensure all important pages are easily discoverable.
- Ensure your robots.txt file is not blocking important crawlers or content.
Recently Updated Pages Needing Re-Crawl Pages edited in the last 60 days where the selected bot either has never crawled them, or the most recent crawl predates your latest edit. These are the pages most likely to show stale content in search/AI results.
5 of 41 page(s) modified in the last 60 days need Googlebot attention: 2 never crawled, 3 crawled before the latest edit.
| Title | Type | Last Modified | Last Crawl | Status & Actions |
|---|---|---|---|---|
| Travel Humidor Buying Guide | Post | 2026-09-28 | Never | never crawled Edit View |
| How to Season a New Humidor | Post | 2026-09-30 | 2026-09-12 | stale · last crawl 23d ago Edit View |
| Cigar Humidity Chart (69% vs 72%) | Post | 2026-10-01 | 2026-09-24 | stale · last crawl 11d ago Edit View |
| Boveda vs Humidity Beads: Which Lasts Longer? | Post | 2026-09-19 | Never | never crawled Edit View |
| Humidor Seasoning FAQ | Page | 2026-09-26 | 2026-09-20 | stale · last crawl 15d ago Edit View |
Pages flagged "never crawled" have no log entry for the selected bot at all. Pages flagged "stale" were crawled, but the most recent crawl predates your last edit. For either case, consider submitting the URL via Google Search Console / Bing Webmaster Tools, or improving internal links to it.
Understanding Days Since Last Crawl Guide to crawl recency and deindex risk levels.
The longer a page goes without being crawled (or re-crawled) by a search engine bot like Googlebot, the higher the potential risk that it might be considered stale or less important, which could eventually affect its indexing status. This report helps identify such pages.
General Guideline (varies by site & bot):
- Low Risk (< 100 days): Page is being crawled regularly. No action needed.
- Medium Risk (100-130 days): Monitor closely, consider updating content or improving internal links.
- High Risk (131-150 days): Higher attention needed. Update content and request re-indexing.
- Critical Risk (151+ days or Never): These might be at greater risk of deindexing if they are important pages.
Content Status by Last Crawl Date Range Distribution chart showing how many pages fall into each crawl recency bucket.
Pages by Days Since Last Crawl (Selected Bot & Risk Level) Detailed table of pages with their last crawl date and risk level.
| Title | Type | Last Crawl | Days Since | Risk Level | Actions |
|---|---|---|---|---|---|
| Spanish Cedar Tray Humidor Review | post | 2026-04-02 | 186 | Critical | Edit |
| Humidor Hygrometer Calibration (Salt Test) | post | 2026-05-01 | 157 | Critical | Edit |
| Cigar Storage Without a Humidor | post | 2026-05-18 | 140 | High | Edit |
| Acrylic vs Wood Humidors | post | 2026-06-12 | 115 | Medium | Edit |
| Our Story | page | Never | Never | Critical | Edit |
| Best Cigar Humidors of 2026 | post | 2026-10-05 | 0 | Low | Edit |
Google Search Console Insights Aggregate report from URLs you have inspected via Google Search Console. Shows indexing verdict mix, coverage states, canonical mismatches, robots.txt blocks, and pages where Google's last crawl predates your last edit. Use Bulk Inspect to grow the dataset — Linkilo throttles requests to stay under Google's 600/min limit.
Which of your pages Google has actually indexed — verdicts, coverage states, canonical mismatches and re-inspection — now lives in the dedicated Index Coverage report. That keeps this analyzer focused on crawl behavior (which bots hit what, and how recently). So far 286 URLs inspected, 241 indexed.
Open Index Coverage →Crawl Waste Analysis Crawls that did not help SEO/AI visibility. Trend arrows compare this window to the previous equal-length window. The "Fix top 3" estimate sums the hits in your three biggest fixable categories. Results are cached for 10 minutes — click Refresh to force a recompute. Computed in 1.4 s · cached for 10 minutes — refresh to recompute.
Where bots spent your crawl budget How Googlebot distributed its crawls across content categories on your site. Big slices outside "Content Pages" usually mean crawl budget is being eaten by archives, parameter URLs, errors, or media.
Efficiency 77% — 19,842 crawls, 4,566 wastedDaily waste rate (last 30 days) Percentage of each day's bot crawls that fell into a waste category. Watch for sustained climbs — they usually mean a parameter explosion or a new redirect chain has started.
Top 3 fixes (ranked by impact)
Update internal links to point at the final URL. 2 chains are three hops deep, starting at /old-category/humidors/.
Redirect /old-category/humidors/ to /cigar-humidors/ or return 410 so bots stop retrying it.
Block sort and search parameters in robots.txt. They produce the same content as the canonical page.
Rules already present in your existing robots.txt have been excluded. Review carefully before adding — overly aggressive rules can hide real content from Google.
User-agent: * Disallow: /*?orderby= Disallow: /*?s= Disallow: /wp-json/
Full breakdown
Hard waste — fix urgently
Bots followed a redirect instead of landing on the page. Each hop is a wasted request.
Suggested fix: Update the internal links that still point at the old URL.
- https://yoursite.com/old-category/humidors/ (684 hits)
- https://yoursite.com/shop/boveda-84/ (212 hits)
Requests that returned 404/410. Bots keep retrying these for weeks.
Suggested fix: Redirect them to the closest live page or return 410 Gone.
- https://yoursite.com/old-category/humidors/ (1104 hits)
Server errors. Googlebot slows its crawl rate when it sees these.
Suggested fix: Check the PHP error log for the timestamps in the Logs tab.
Probable waste — usually worth fixing
Sort, filter and search parameters that duplicate a canonical page.
Suggested fix: Disallow the parameter patterns in robots.txt and set canonicals.
- https://yoursite.com/?s=humidor&orderby=price (866 hits)
Archive pages with little unique content.
Suggested fix: Noindex archives you do not need in search, or trim pagination depth.
- https://yoursite.com/2024/03/ (118 hits)
- https://yoursite.com/author/editor/page/9/ (64 hits)
wp-json and RSS endpoints fetched by crawlers.
Suggested fix: Disallow /wp-json/ for crawlers that do not need it.
- https://yoursite.com/wp-json/wp/v2/posts?page=14 (540 hits)
Configuration waste — site setting
The same page fetched under two spellings of its URL.
Suggested fix: Enforce one canonical form with a single 301 at the server level.
- https://yoursite.com/Best-Cigar-Humidors-2026 (212 hits)
Sitemap Health (Sitemap vs. Crawled) Pages listed in your sitemap that Googlebot has not crawled in the last 30 days (often: thin content, low link equity, or sitemap noise). Plus URLs Googlebot crawled successfully that are NOT in your sitemap (orphan content that should be added).
In sitemap but uncrawled (top 25)
Pages your sitemap claims matter, that Googlebot has not actually fetched in 30 days. Often: thin content, deep pagination, or low link equity.
- https://yoursite.com/humidor-glossary/
- https://yoursite.com/cigar-cutters-guide/
- https://yoursite.com/about/press/
Crawled but missing from sitemap (top 25)
Real-looking content URLs Googlebot is finding via internal links, but your sitemap does not list. Add the legitimate ones to the sitemap so they get prioritized.
- https://yoursite.com/humidor-seasoning-faq/
- https://yoursite.com/tag/boveda/
Response Time Analytics Server response times broken down by bot and URL. Identify slow pages that waste crawl budget.
By Bot
| Bot | Avg | Min | Max | Requests |
|---|---|---|---|---|
| Googlebot | 262ms | 48ms | 1840ms | 9,411 |
| Bingbot | 301ms | 52ms | 2210ms | 3,122 |
| GPTBot | 612ms | 90ms | 4120ms | 2,804 |
| ClaudeBot | 388ms | 71ms | 2980ms | 1,642 |
| PerplexityBot | 344ms | 66ms | 2450ms | 1,311 |
Trend
Slowest Pages
| URL | Avg Time | Hits |
|---|---|---|
| yoursite.com/old-category/humidors/ | 1204ms | 1230 |
| yoursite.com/?s=humidor&orderby=price | 980ms | 866 |
| yoursite.com/best-cigar-humidors-2026/comment-page-3/ | 844ms | 212 |
| yoursite.com/wp-json/wp/v2/posts?page=14 | 388ms | 540 |
Bot Behavior Analytics Patterns in how each crawler navigates your site — frequency, preferred paths, and activity timing.
Crawl Activity by Hour
Content Type Distribution
Active Bots (Last 7 Days)
Crawl Insights (LLM Era) Modern crawl signals that matter for AI search visibility: how AI bots are split between training/search/user-fetched, mobile vs desktop Googlebot, 304-Not-Modified efficiency, and any bot showing an unusual spike or drop today.
AI bot traffic by purpose (last 7 days)
Googlebot: mobile vs desktop
Mobile-first crawling, as expected.
304 Not Modified efficiency
Healthy use of conditional requests — your server saves work on unchanged pages.
Today's anomalies (vs 7-day average)
- 📈 GPTBot — today 612 hits vs 7-day avg 400 (1.5×)
Verified vs Spoofed Bot Traffic Anyone can put "Googlebot" in their User-Agent. Linkilo verifies each source IP via reverse DNS + forward-confirm — exactly the procedure Google, Bing, OpenAI and Anthropic publish for their bots. "Spoofed" rows are someone pretending to be that bot. Results are cached for 30 days; the first load is slow as new IPs are checked.
99% of crawl hits in the last 7 days came from verified sources. 142 hits were from spoofers impersonating real bots.
| Bot | Verified IPs | Spoofed IPs | Unknown | Verified Hits | Sample Spoofers |
|---|---|---|---|---|---|
| Googlebot 9411 hits / 41 IPs | 38 | 3 | 0 | 99% | 203.0.113.17, 198.51.100.9 |
| Bingbot 3122 hits / 22 IPs | 21 | 1 | 0 | 98% | 192.0.2.44 |
| GPTBot 2804 hits / 17 IPs | 17 | 0 | 0 | 100% | — |
| ClaudeBot 1642 hits / 9 IPs | 9 | 0 | 0 | 100% | — |
Crawl Logs Raw server log entries showing every crawl request with full details.
| Time | Bot | URL | Status | Response |
|---|---|---|---|---|
| Oct 5, 09:42 AM | googlebot | /best-cigar-humidors-2026/ | 200 | 214ms |
| Oct 5, 09:41 AM | gptbot | /cigar-humidity-chart-69-vs-72/ | 200 | 598ms |
| Oct 5, 09:41 AM | googlebot | /old-category/humidors/ | 404 | 1188ms |
| Oct 5, 09:39 AM | bingbot | /shop/boveda-84/ | 301 | 96ms |
| Oct 5, 09:38 AM | perplexitybot | /how-to-season-a-new-humidor/ | 200 | 338ms |
| Oct 5, 09:36 AM | claudebot | /travel-humidor-buying-guide/ | 200 | 402ms |
| Oct 5, 09:35 AM | googlebot | /wp-json/wp/v2/posts?page=14 | 200 | 384ms |
| Oct 5, 09:33 AM | oai-searchbot | /humidor-seasoning-faq/ | 304 | 61ms |
| Oct 5, 09:31 AM | googlebot | /?s=humidor&orderby=price | 500 | 2140ms |
Save a snapshot Captures total crawls, unique URLs, status-code distribution, bot mix, top crawled URLs, coverage by bot, health score, crawl waste, and GSC summary at this moment in time.
Saved snapshots Manual snapshots stay until you delete them. Automated weekly snapshots keep a rolling history. Use View to open the printable report, Compare to diff against another snapshot, CSV to export raw numbers, or Delete to remove an obsolete snapshot.
Quick: open a live report (no save)
Generate a printable report from the current numbers without saving a snapshot. Useful for ad-hoc client emails.
Download 7-day CSVAlert Configuration Set up automated notifications when crawl metrics exceed your thresholds.
| Alert Name | Type | Conditions | Recipients | Status | Actions |
|---|---|---|---|---|---|
| Googlebot Inactivity Alert | No Crawl Activity | No Googlebot crawl for 24 hours | Admin | Active | |
| Error spike | High Error Rate | Error rate > 5% | Admin + Custom | Active | |
| GPTBot slow responses | Slow Response | Response time > 1000ms | Admin | Inactive |
What Add Alert opens
Understanding Web Crawlers & AI Bots A reference guide to the different types of bots that visit your site and how to manage them.
Web crawlers are automated programs that visit websites to index content, gather data, or generate previews. Understanding which bots visit your site helps you optimize for search engines, manage AI training opt-outs, and ensure proper social media sharing.
Search Engine Crawlers Crawlers from major search engines that index your content for search results.
These bots index your content for search results. Blocking them removes your site from their search.
| Bot | Company | Purpose | Recommendation |
|---|---|---|---|
| Googlebot | Primary crawler for Google Search. Essential for visibility. | Always Allow | |
| Bingbot | Microsoft | Indexes for Bing Search and powers Copilot features. | Always Allow |
| DuckDuckBot | DuckDuckGo | Privacy-focused search engine with growing user base. | Recommended |
| YandexBot | Yandex | Leading search engine in Russia and CIS countries. | Regional |
| Baiduspider | Baidu | China's largest search engine. | Regional |
| Applebot | Apple | Powers Siri and Spotlight search suggestions. | Recommended |
AI/LLM Training Crawlers Bots that collect content for AI model training. You can safely block these without affecting search rankings.
These bots collect content for AI training. You can block them via robots.txt without affecting search rankings.
Tip: Blocking these will NOT affect your Google/Bing search rankings.
| Bot | Company | Purpose | Traffic Level |
|---|---|---|---|
| GPTBot | OpenAI | Collects data for GPT models. Highest volume AI crawler (~30% of AI traffic). | Very High |
| Google-Extended | For Gemini AI training. Separate from Googlebot - safe to block. | High | |
| ClaudeBot | Anthropic | Trains Claude AI models. Respects robots.txt. | Medium |
| Meta-ExternalAgent | Meta | For Meta's Llama AI models (~19% of AI traffic). | Very High |
| PerplexityBot | Perplexity | Fastest growing AI crawler (157,000%+ growth in 2024). | High |
| CCBot | Common Crawl | Open datasets used by many AI companies. | Medium |
| Bytespider | ByteDance | TikTok parent company's AI crawler. | Declining |
| Grok | xAI | Trains Grok AI for X (Twitter). | Low-Medium |
AI Assistant Crawlers (User-Triggered) Bots that fetch your content when AI users request information. These can generate referral traffic.
These fetch pages when users ask AI assistants questions. They can drive referral traffic when AI cites your content.
Recommendation: Consider ALLOWING these - they represent real users and can send traffic to your site.
| Bot | Company | Purpose | Referral Potential |
|---|---|---|---|
| ChatGPT-User | OpenAI | Fetches pages when ChatGPT users ask questions (2,825%+ growth). | High |
| Claude-User | Anthropic | Fetches pages when Claude users request web content. | Medium |
| Perplexity-User | Perplexity | Fetches for search answers. Note: May sometimes ignore robots.txt. | High |
| OAI-SearchBot | OpenAI | OpenAI's dedicated search crawler for ChatGPT search. | Medium-High |
Social Media Crawlers Bots from social platforms that generate link previews when your content is shared.
Generate link previews when URLs are shared. Blocking them breaks previews.
| Bot | Platform | Purpose | Recommendation |
|---|---|---|---|
| facebookexternalhit | Meta | Link previews for Facebook and Instagram. | Always Allow |
| Twitterbot | X (Twitter) | Twitter/X card previews. | Always Allow |
| LinkedInBot | Link previews for LinkedIn posts. | Always Allow | |
| Slackbot | Slack | Link previews in Slack workspaces. | Recommended |
| Discordbot | Discord | Link previews in Discord. | Recommended |
SEO Tool Crawlers Crawlers from SEO tools that analyze your site. Can be resource-intensive.
Gather data for SEO analysis. Can be resource-intensive. Consider rate-limiting if performance issues occur.
| Bot | Tool | Purpose | Activity Level |
|---|---|---|---|
| AhrefsBot | Ahrefs | Backlink analysis and site auditing. | Very High |
| SemrushBot | Semrush | Keyword and competitor research. | High |
| MJ12bot | Majestic | Link intelligence database. | Medium |
| DotBot | Moz | Domain authority calculations. | Low |
Quick robots.txt Reference Common robots.txt rules for allowing or blocking specific crawlers.
Your robots.txt: https://yoursite.com/robots.txt
Block AI Training
User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: meta-externalagent Disallow: /
Allow AI Assistants
User-agent: ChatGPT-User Allow: / User-agent: Claude-User Allow: / User-agent: Perplexity-User Allow: / User-agent: OAI-SearchBot Allow: /
Note: Not all bots respect robots.txt. For stronger control, use .htaccess rules or a WAF.
Recreated from the plugin’s own template and stylesheet. Same panels, same labels.
Linkilo vs Screaming Frog Log File Analyser, feature by feature
Capabilities as of October 2026, from the Log File Analyser’s user guide and release notes for version 7.1. Both tools are good at what they are for.
| Capability | Linkilo | Screaming Frog Log File Analyser |
|---|---|---|
| Needs raw server log files | No | Yes; Apache, NGINX, IIS, ELB, HAProxy, JSON, CSV |
| Records continuously | Yes | No; import when you have logs |
| Sees cached and CDN-served hits | No | Yes |
| Works on non-WordPress sites | No | Yes |
| Bot verification | Published IP ranges, reverse DNS with forward confirmation | IP lists, reverse DNS, ASN and custom rules per user agent |
| AI crawlers tracked by name | GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, CCBot and more, 40+ bots in all | Eight presets plus custom user agents; AI bot verification since 7.0 |
| Pages not crawled lately, ranked | Yes | Days-since-last-crawl column |
| Edited pages Google has not returned to | Yes | No |
| Crawl waste by URL pattern with a robots.txt block | Yes | Directories and URL tabs; no robots output |
| Google’s index verdict per URL | URL Inspection, in place | No |
| Sitemap vs crawl comparison | Yes | Import a sitemap or crawl export |
| Orphans by comparing a crawl to the log | Orphans from the link index, with suggested sources | Yes, with a Spider export |
| Alerts when a bot goes quiet or errors spike | Email, hourly check | No |
| Scheduling or automation | Always on | None; no scheduler, CLI import or API |
| Scale | Bounded table with configurable retention | Millions of lines, disk-bound |
| Time series and period comparison | Snapshots with change arrows | Time Series tab since 7.0 |
| Exports | PDF and CSV, ready to send | CSV, Excel, Google Sheets; graphs not exportable |
| Fix what the log reveals | Broken links, redirects and orphans, in the same plugin | No; hands off to the SEO Spider |
| Claude / ChatGPT connection (MCP) | Yes | No |
| Priced per | Site | User |
Compiled from each product’s own public documentation. If something here is out of date about Screaming Frog Log File Analyser, tell us and we will correct it.
Per site against per user
Different units, so read the row that matches how you work. Screaming Frog is in pounds because that is how it is sold.
| You are | Linkilo | Screaming Frog Log File Analyser |
|---|---|---|
| One person, one site | $59 / year | £99 / yearor free up to 1,000 log lines |
| One person, ten sites | $279 / year | £99 / yearunlimited projects per licence |
| A team of five, unlimited sites | $499 / year | £89 per user / yearfive to nine licences |
| Renewal | Yearly; keeps working if it lapses | Yearly; reverts to the free limits on expiry |
| Money-back window | 30 days | 14 days |
| Also needed | Nothing | A host that gives you raw logs; the SEO Spider (£199 a year) for crawl comparisons |
- An analyst with many sites and good access to logs gets more from the Screaming Frog model. A site owner or an agency that runs the sites gets more from the per-site model.
- The crawl log is one Linkilo module. The same licence covers suggestions, broken links, redirects, clusters and the rest.
Screaming Frog Log File Analyser prices and terms as published on its site on 16 October 2026. Linkilo prices are the current list prices. If either has moved, tell us.
Where Screaming Frog Log File Analyser is the better choice
No tool wins at everything, and a comparison page that pretends otherwise is not worth reading.
For a log analyst, Screaming Frog is the better tool and Linkilo is not trying to be it. It reads what the server saw, which includes every hit your page cache, CDN or static file server answered before WordPress ran, and on a heavily cached site that is a large share of bot traffic Linkilo never sees. It handles millions of lines, verifies bots by IP list, reverse DNS, ASN or your own rules, and since version 7.0 verifies the AI crawlers too. Import a Spider crawl or a sitemap and it finds orphans by set difference, which is the textbook method. It works on any platform, keeps the data on your machine, and backfills three months of history from logs you already have on day one. It is also cheap for what it is, and the free tier is enough to learn on. If your job is auditing sites from their logs, buy it.
Your site is heavily cached or behind a CDN
Bot hits the cache answers never reach WordPress. Screaming Frog sees them in the server or CDN log; Linkilo does not.
You audit sites you do not run
A desktop tool that takes a log export works on a client site, a Shopify store or a static site. Linkilo has to be installed on a WordPress site.
You want history from before today
Import three months of existing logs and start with a backfill. Linkilo records from the moment it is activated.
You are already a Screaming Frog shop
The Spider and the Log File Analyser are built to be used together, and the crawl-versus-log workflow is the best orphan method there is.
Where Linkilo pulls ahead
No logs to fetch
Managed hosts ration raw logs or do not expose them. Linkilo records each bot visit itself, so the monthly download-and-import ritual does not exist.
See howIt is on all the time
A new post was published this morning. Was it fetched? Linkilo knows by lunch. A desktop import knows at the end of the month, if you remembered to run it.
See howIt tells you when something changes
Email alerts when Googlebot goes quiet, errors spike or a bot starts hammering you, with anomaly detection on top. Screaming Frog has no scheduler and no alerts.
See howIt knows which post that URL is
A log line is a path. Linkilo joins it to the post, its inbound links, its cluster and its last edit, so a stale page is a page you can act on, not a string to look up.
See howIt asks Google directly
Click Inspect on any URL and see Google’s own verdict: indexed, not indexed, blocked or canonicalised elsewhere, without leaving WordPress.
See howThe fix is one click away
The log shows bots hitting 404s and redirect chains. The same plugin repairs the links, flattens the chains and suggests sources for the orphans. Screaming Frog hands you off to another tool.
See howFrom a monthly import to a live log
There is nothing to migrate, because the data was never in a plugin. Here is what the first month looks like, and what to keep Screaming Frog for.
Activate and let it run
Logging starts at activation. Each bot visit is one lightweight database insert, nothing runs for human visitors, and retention is configurable so the table stays bounded.
Give it a week
Linkilo records from day one and has no backfill. After a week you have enough to see crawl frequency by section, the first stale pages and which AI crawlers actually visit.
Check what your cache hides
If you run a full-page cache or a CDN, compare a day of Linkilo’s log with the CDN dashboard once. The difference is the share of bot traffic that never reaches WordPress; knowing it keeps the numbers honest.
Connect Search Console
This switches on URL Inspection in place, so a stale page can be checked against Google’s own index status without leaving the screen.
Set the alerts
Googlebot gone quiet, error rate up, a bot hammering you. The check runs hourly and emails you. This is the part the desktop workflow cannot do at all.
Keep Screaming Frog for the deep audit
A quarterly import of raw logs into the Log File Analyser still catches the cached hits and gives you the crawl-versus-log orphan check. The two are a good pair; Linkilo is the one that is on every day.
Which one is right for you
Pick Screaming Frog if
You audit from logs, across platforms, and have the raw files.
With Linkilo: nothing matches it for depth, and the free tier gets you started.
Pick Linkilo if
Your host will not give you raw logs, or you are tired of asking.
With Linkilo: the site records its own crawl log and you never download a file.
Pick Linkilo if
You want to know today, not at the end of the month, whether Google and the AI crawlers fetched a page.
With Linkilo: live logging with alerts when a bot goes quiet.
Pick Linkilo if
Reading the log is only useful if someone then fixes the links, and that someone is you.
With Linkilo: the repair tools are in the same plugin as the log.
Questions people ask before choosing
No. It records bot visits itself as WordPress serves the page. There is nothing to download, upload or import, and nothing leaves your site.
Requests answered before WordPress runs: a full-page cache hit, a CDN edge hit, a static file. On a lightly cached site that is a small share; on a heavily cached site it can be most bot traffic. The honest answer is to compare a day against your CDN dashboard once and know the ratio.
Each major bot is checked against its published IP ranges, with a reverse DNS lookup and forward confirmation for the ones that publish hostnames instead. Verified and spoofed visits are kept apart so impostors never pollute the numbers.
GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, CCBot, Amazonbot, Applebot and the rest of the forty-plus bots in its list, each by name, each verified where the operator publishes ranges.
A bot visit costs one lightweight insert, under a hundredth of a second, and nothing runs for human visitors. Retention is configurable and old rows are pruned automatically, so the table is bounded rather than growing forever.
No. Linkilo records from activation. If you have old raw logs and want the history, that is a job for Screaming Frog, and the two are not mutually exclusive.
Yes. Save a snapshot, compare any two periods with change arrows, and export a PDF or CSV that is ready to send without editing.
Differently. Screaming Frog compares a crawl export with the log. Linkilo already holds the link index, so it knows which posts have no inbound links without a crawl, suggests posts that should link to them, and shows whether bots have been reaching them anyway.
If you have the logs and the licence, yes, for the quarterly deep audit and the cached hits. Linkilo is the one that is on every day and tells you when something changes.
Link Whisper vs Linkilo
The closest alternative. Here is what each one does, side by side, without the marketing.
Interlinks Manager vs Linkilo
Interlinks Manager is being retired for Link Manager. Render-time rules versus links that stay in your posts.
Internal Link Juicer vs Linkilo
Automatic keyword linking versus reviewed, explainable suggestions.
Every Linkilo feature
The full list, grouped by the problem each part solves.
Find out who crawled your site this morning.
Activate Linkilo, give it a week, and read the crawl log against posts you know. No log files, no desktop app, and a 30-day money-back guarantee.
30-day money-back guarantee. Every feature is included in every plan.