Linkilo Crawler Analysis: Complete User Guide

Go Back to Main Knowledgebase

What is the Crawler Analyzer?

The Crawler Analyzer is a feature in Linkilo that shows you which bots (automated programs) are visiting your website. Think of it like a visitor log, but specifically for robots like Google, Bing, ChatGPT, and other services.

Why does this matter? Understanding bot traffic helps you see if search engines are finding your content, if AI companies are scraping your site, and if there are any technical problems.

Where to Find It

In your WordPress dashboard, look for:

1. Linkilo in the left sidebar
2. Click on Crawler Analyzer

Understanding the Dashboard

When you open the Crawler Analyzer, you’ll see several sections. Here’s what each one means:

Health Score

This is your overall “grade” for how well bots can access your site. A high score (80-100) means everything is working great. A low score means there might be issues to fix.

Quick Action Cards

At the top, you’ll see three clickable cards:

• Issues: Problems that need your attention
• Uncrawled: Pages that haven’t been visited by bots recently
• Errors: Pages returning error codes (like 404 “not found”)

Tip: Click any card to see more details about that specific issue.

The Tabs Explained

The dashboard has several tabs. Here’s a simple explanation of each:

Tab	What It Shows
Overview	A summary of all bot activity: total visits, success rate, errors, and response times. Start here for a quick snapshot.
Health	Your site’s crawl health score with a breakdown of what’s good and what needs fixing.
Coverage	Shows what percentage of your site each bot has visited. Useful to see if Google is finding all your pages.
Days Since Crawl	Lists pages that haven’t been crawled recently. Pages not visited for 30+ days may be at risk of losing their search rankings.
Crawl Budget	Shows if bots are wasting time on unimportant pages instead of your key content. A high efficiency score is good.
Response Times	How fast your site responds to bot requests. Slower pages may rank lower in search. Under 500ms is ideal.
Bot Behavior	Shows when bots visit (by hour) and what types of content they access most.
Logs	A detailed list of every bot visit. You can filter by bot type, status code, date, or search for specific URLs.
Alerts	Set up email notifications for important events, like if Google stops crawling your site.
Bot Reference	A helpful guide explaining what each bot does and whether you should allow or block it.

Understanding Bot Types

Not all bots are the same. Here are the main categories:

🔍 Search Engine Bots

Examples: Googlebot, Bingbot, DuckDuckBot

What they do: Index your pages so they appear in search results

Recommendation: Always allow these! Blocking them removes you from search.

🤖 AI Training Bots

Examples: GPTBot, ClaudeBot, Google-Extended, Meta-ExternalAgent

What they do: Collect content to train AI models

Recommendation: Your choice! Blocking these won’t affect your search rankings.

💬 AI Assistant Bots

Examples: ChatGPT-User, Claude-User, Perplexity-User

What they do: Fetch your pages when real users ask AI assistants questions

Recommendation: Consider allowing – these can send traffic to your site when AI cites your content!

📱 Social Media Bots

Examples: Facebook, Twitterbot, LinkedInBot, Slackbot

What they do: Generate link previews when your URLs are shared

Recommendation: Always allow! Blocking breaks your link previews on social media.

📊 SEO Tool Bots

Examples: AhrefsBot, SemrushBot, MJ12bot

What they do: Gather data for SEO analysis tools

Recommendation: Usually fine to allow, but you can block if they cause performance issues.

Setting Up Alerts

Alerts notify you by email when something important happens. Here’s how to set one up:

1. Go to the Alerts tab
2. Click Add Alert
3. Choose an alert type:

• No Crawl Activity: Get notified if a specific bot (like Google) hasn’t visited in X hours
• High Error Rate: Get notified if too many pages are returning errors

1. Give your alert a name and set the conditions
2. Click Save

Suggested Alert: Create a “No Crawl” alert for Googlebot set to 48 hours. This way, you’ll know quickly if Google stops visiting your site.

You can also go to the Log tab and select different status codes and bots to see what URLs have been affected.

Common Questions

Q: What’s a good health score?

A: Aim for 80 or higher. A score of 90+ is excellent. If you’re below 70, check the Health tab for specific issues to fix.

Q: Should I worry if AI bots are crawling my site?

A: It depends on your preference. AI training bots collect your content, but blocking them won’t hurt your search rankings. The Bot Reference tab has more details.

Q: Why are some of my pages “uncrawled”?

A: New pages take time to be discovered. For older pages, check if they’re linked properly from your sitemap and other pages. Orphaned pages are harder for bots to find.

Q: What do the status codes mean?

• 200: Success – page loaded correctly
• 301/302: Redirect – page moved to a new URL
• 404: Not Found – page doesn’t exist
• 500: Server Error – something went wrong on your server

Quick Tips for Success

• Check weekly: A quick glance at the Overview tab once a week keeps you informed
• Fix errors first: 404 and 500 errors should be your top priority
• Speed matters: If response times are over 1 second, consider optimizing your site
• Use alerts: Set up at least one alert for Googlebot so you know if something breaks
• Export your data: Use the Export button to save reports for your records

—

Need more help? Visit the Bot Reference tab for detailed information about each crawler.