Linkilo Crawler Analysis: Complete User Guide

Go Back to Main Knowledgebase

Welcome to the Linkilo Crawler Analysis tool — your window into how search engines and AI bots interact with your website. This powerful tool tracks and analyzes crawler behavior to help you improve your site’s visibility, fix technical issues, and ensure your content is being properly discovered and indexed. This guide will walk you through each section of the tool and explain how to use the data to optimize your site’s performance.

Dashboard

What is the Dashboard?

The Dashboard is your central command center for monitoring how search engine crawlers and AI bots interact with your website. It provides a quick overview of your site’s crawler activity with real-time metrics and visualizations.

Key Features:

Today’s Crawls

This metric shows you how many times search engines and AI bots have visited your site today. A healthy number indicates your site is being regularly discovered and indexed.

Success Rate

This percentage shows how many crawler requests were successful (typically 200 status codes). A high success rate indicates your site is responding well to crawler requests, which is crucial for proper indexing.

Errors Today

This counter tracks how many errors crawlers encountered while trying to access your pages today. Monitoring this helps you quickly identify and fix issues that might prevent proper indexing.

Agent Activity Chart

The 7-day trend chart shows you which search engines and AI tools are most actively crawling your site and how their activity has changed over the past week. Use this to identify patterns or sudden changes in crawler behavior.

Status Code Distribution

This pie chart visualizes the distribution of HTTP status codes returned when crawlers access your site. Ideally, most responses should be 200 (success). A significant number of 404s (not found) or 5xx (server errors) indicates problems that need attention.

Most Frequently Visited URLs

This list shows which pages on your site receive the most crawler attention. Pages frequently visited by crawlers are typically considered important by search engines, so this list gives you insight into what content search engines find most valuable on your site.

How to Use the Dashboard Effectively:

1. Check it daily: Make a quick daily check to catch any sudden spikes in errors or drops in crawler activity.
2. Filter by specific bots: Use the dropdown to focus on specific search engines or AI tools. For example, filtering for Google will show you only Googlebot activity, which is particularly important for Google search rankings.
3. Identify trends: Look for patterns in crawler behavior. If you notice a specific search engine hasn’t visited in days, you might have issues with that particular engine.
4. Take action on anomalies: If you spot unusual activity – like a sudden drop in crawls or spike in errors – investigate immediately using the other tabs.

Coverage Insights

What are Coverage Insights?

Coverage Insights show you how thoroughly search engines and AI bots are discovering and crawling your website’s content. This section helps identify content that’s being missed by crawlers, which could limit your visibility in search results and AI tools.

Key Features:

Total Site Pages

This counter shows the total number of published posts, pages, and custom post types on your site. This serves as the baseline for measuring crawler coverage.

Crawler Coverage Table

This comprehensive table shows:

• Which crawlers are visiting your site
• How many unique URLs each crawler has accessed
• What percentage of your total content each crawler has discovered
• A visual representation of the coverage (with color-coding: red for low, yellow for medium, blue for good coverage)

Crawler Coverage Comparison Chart

This chart provides a visual comparison of how different search engines and AI tools are covering your site. It helps you identify which crawlers might be missing significant portions of your content.

Recently Updated Pages Not Crawled

This section lists pages that have been recently updated but haven’t been visited by the selected crawler. This is crucial for identifying content that should be in search indexes but might be invisible because crawlers aren’t finding it.

How to Use Coverage Insights Effectively:

1. Identify coverage gaps: Look for crawlers with particularly low coverage percentages. If important search engines like Google are missing a significant portion of your content, this needs immediate attention.
2. Focus on recently updated content: Check if your latest content or important updates are being discovered by crawlers. If not, you may need to improve your internal linking or submit URLs directly to search engines.
3. Compare crawler behaviors: Different search engines may discover different parts of your site. Understanding these patterns can help you optimize for specific search engines.
4. Take action on uncrawled pages: For important pages that aren’t being crawled:
   • Improve internal linking to these pages
   • Add them to your XML sitemap
   • Use the “Actions” buttons to submit URLs directly to search engines
5. Monitor changes over time: Regularly check if your coverage is improving after implementing changes to your site structure or content.

Crawl Analysis

What is Crawl Analysis?

Crawl Analysis provides detailed logs of every crawler visit to your website. This granular data helps you understand exactly how search engines and AI tools are interacting with your site, which pages they’re accessing, and any issues they encounter.

Key Features:

Detailed Crawler Logs

The main table displays comprehensive information about each crawler visit:

• Time of visit
• Which crawler (agent) accessed your site
• The exact URL that was accessed
• The HTTP status code returned
• How long it took your server to respond (response time)

Advanced Filtering

You can filter the log data by:

• Specific crawler (e.g., Googlebot, Bingbot, ChatGPT, Claude)
• Date range (from/to)

How to Use Crawl Analysis Effectively:

1. Investigate specific issues: If you notice problems in other sections, use Crawl Analysis to dig deeper. For example, if you see a spike in errors on the Dashboard, filter the logs to that time period to find the exact pages causing problems.
2. Monitor specific pages: If you’ve just published important content, check if and when crawlers are discovering it.
3. Identify slow-loading pages: Look for URLs with long response times. Search engines factor page speed into rankings, so pages that consistently load slowly for crawlers may not rank as well.
4. Track specific crawlers: Filter by a specific crawler to see exactly what content different search engines or AI tools are accessing. This is particularly useful if you’re focusing on improving visibility in a specific search engine.
5. Identify crawl patterns: Look for patterns in how frequently specific sections of your site are crawled. This can reveal what content search engines consider most important on your site.
6. Spot crawler traps: If you see the same crawler repeatedly hitting similar URLs with errors, you might have a “crawler trap” (a structure that causes crawlers to get stuck in loops).

Status Codes

What are Status Codes?

The Status Codes section provides focused insights into specific HTTP response codes that indicate problems or redirects on your site. Monitoring these codes helps you identify and fix issues that could affect your search visibility.

Key Features:

404 Errors List

This list shows all URLs that returned a 404 (Not Found) status when crawlers attempted to access them. These are potentially harmful to your SEO as they:

• Waste your crawl budget
• Create dead ends for both users and search engines
• May indicate broken internal links or external sites linking to deleted content

301 Redirects List

This list shows all URLs that are redirecting to other locations (301 Permanent Redirect). While redirects are normal, especially after site restructuring, excessive redirects can:

• Slow down crawlers
• Dilute link equity
• Create potential redirect chains that confuse search engines

How to Use Status Codes Effectively:

1. Fix broken links: For each 404 error, determine the source and either:
   • Restore the missing content
   • Create a proper 301 redirect to the relevant replacement page
   • Fix internal links pointing to the broken URL
2. Optimize redirects: Review your 301 redirects to ensure they’re necessary and properly implemented:
   • Eliminate redirect chains (A→B→C)
   • Update internal links to point directly to final destinations rather than through redirects
   • Consider consolidating similar redirects
3. Investigate patterns: Look for patterns in the errors, such as:
   • Multiple 404s in a specific section of your site
   • Errors in a particular URL structure
   • Redirects that could be eliminated with a site-wide rule
4. Check external links: If you see 404 errors for URLs you don’t recognize, check if external sites are linking to non-existent content on your domain. These represent opportunities to reclaim lost link value with proper redirects.
5. Monitor after site changes: After making significant changes to your site structure, regularly check this section to ensure you haven’t introduced new errors or unnecessary redirects.

Settings

What are the Settings?

The Settings section allows you to customize how the Linkilo Crawler Analysis tool works with your site. These configurations help you focus on the data that matters most to you and manage system resources efficiently.

Key Features:

Data Retention

This setting controls how long crawler data is stored in your database. You can choose between:

• 7 days (smallest database footprint, good for high-traffic sites)
• 30 days (balanced option for most sites)
• 90 days (best for long-term trend analysis, requires more database space)

Track These Crawlers

This section lets you select which specific crawlers you want to monitor. Options include:

Search Engine Crawlers:

• Googlebot
• Bingbot
• YandexBot
• Baiduspider
• DuckDuckBot
• Yahoo Slurp
• AppleBot

AI Crawlers:

• Grok (AI)
• ChatGPT (AI)
• Claude (AI)

How to Use Settings Effectively:

1. Optimize for your site size: If you have a smaller site or limited hosting resources, choose a shorter data retention period. Larger sites with more resources might benefit from the longer trend analysis possible with 90-day retention.
2. Focus on relevant crawlers: For most sites, tracking Googlebot, Bingbot, and a few major AI crawlers like ChatGPT and Claude is sufficient. If you target specific international markets, add region-specific crawlers like YandexBot (Russia) or Baiduspider (China).
3. Balance completeness vs. performance: Tracking all crawlers provides the most complete picture but creates more database load. If your site experiences performance issues, consider tracking only the most critical crawlers for your audience.
4. Review periodically: As search technologies evolve, new crawlers may become important. Periodically review your crawler selection to ensure you’re tracking the most relevant bots for your site’s visibility.
5. Save changes: After making any adjustments, be sure to click the “Save Changes” button to apply your new settings.