Crawl Analysis Issues and Best Practices

Go Back to Main Knowledgebase

If you’re working with the Linkilo Crawler Analysis feature and running into issues, this guide will help you identify and resolve common problems.

No Crawl Data Showing

When you open the Crawler Analysis and see no data at all, it’s usually one of a few straightforward issues.

The most common reason is that the feature simply hasn’t had time to collect data yet. The crawler logging needs actual bot visits to your site to record data. If you’ve just enabled the feature, you’ll need to wait for search engines and other bots to visit your site naturally. This typically happens within 24-48 hours for active sites.

To verify if logging is working:

1. Navigate to the Crawler Analysis page
2. Check the “Last Updated” timestamp in the dashboard
3. If it shows “Never”, the system hasn’t recorded any bot visits yet
4. Try expanding your date range to “Last 30 Days” or “Based on Retention Setting”

If you’ve been waiting and still see no data:

First, check that crawler logging is actually enabled. Some users accidentally disable it without realizing. The feature needs to be active to collect data.

Next, verify your bot tracking settings. Go to the Settings tab (if available) or check which bots you’re tracking. By default, only major search engines like Googlebot and Bingbot are tracked. If you’re expecting to see other bots, you’ll need to enable them in the tracking settings.

Database table issues:

Sometimes the database table needed for logging doesn’t get created properly. To fix this:

1. Deactivate the Linkilo plugin
2. Wait a few seconds
3. Reactivate the plugin
4. This should recreate any missing database tables

Inaccurate or Missing Bot Detection

If you’re seeing legitimate bot traffic in your server logs but it’s not appearing in the Crawler Analysis, or if bots are being misidentified, there are several things to check.

Bot identification relies on user agent strings. Some bots may use non-standard user agents that aren’t recognized. The plugin tracks common patterns for major search engines and AI bots, but newer or less common bots might not be detected.

To improve bot detection:

1. Check which bots are enabled for tracking in your settings
2. Make sure “Other Bots” is enabled if you want to catch generic crawlers
3. Understand that some legitimate tools (like uptime monitors) might appear as “Other Bots”

False positives can occur when automated tools or scrapers pretend to be search engine bots. The plugin can’t always distinguish between real Googlebot and something pretending to be Googlebot. This is a limitation of user agent-based detection.

Coverage Percentage Seems Wrong

If the coverage percentages don’t match your expectations, understand how they’re calculated. The plugin compares unique URLs crawled by each bot against your total published content.

Common reasons for unexpected coverage:

Your site might have more published content than you realize. The total includes all published posts, pages, and custom post types. If you have many old posts or forgotten pages, they’re counted in the total even if they’re not linked anywhere.

Some content might be blocked from crawlers. Check your robots.txt file and ensure important content isn’t accidentally blocked. Also verify that your XML sitemap is up to date and submitted to search engines.

Coverage calculation notes:

• A page needs to be crawled at least once to count as “covered”
• The calculation is based on the retention period you’ve set
• If you’ve recently cleared old logs, coverage will appear lower until bots recrawl your content

Performance Issues with Large Datasets

If the Crawler Analysis interface is loading slowly or timing out, you’re likely dealing with too much data at once.

Immediate solutions:

1. Use a shorter time range (7 days instead of 30 or 90)
2. Filter by specific bots rather than viewing “All Agents”
3. Apply URL filters to focus on specific sections of your site

For sites with heavy bot traffic:

Consider adjusting your retention settings to keep less historical data. While 200 days is recommended for the “Days Since Last Crawl” feature, you might need to balance this with performance. 30-90 days is usually sufficient for most analysis needs.

“Days Since Last Crawl” Showing Unexpected Results

This feature tracks when each page was last visited by the selected bot. If the numbers seem off, here’s what to check:

Data retention limits: The feature can only report on data within your retention period. If you have retention set to 30 days, any page not crawled in those 30 days will show as “Never Crawled” even if it was crawled 31 days ago.

To get more accurate results:

1. Increase your retention period to 200 days (recommended)
2. Allow time for data to accumulate after changing the setting
3. Remember that historical data before you enabled the feature won’t be available

Important pages showing as “Never Crawled”:

This is actually valuable information. If important pages haven’t been crawled within your retention period, they need attention. Consider:

• Improving internal linking to these pages
• Submitting them directly through Google Search Console
• Updating the content to trigger a recrawl

Chart Display Issues

If charts aren’t displaying or showing incorrect data:

1. Clear your browser cache – Sometimes old JavaScript gets cached
2. Check browser console for errors (F12 in most browsers)
3. Try a different browser to rule out browser-specific issues
4. Disable browser extensions that might interfere with JavaScript

Charts require JavaScript to function. If you have JavaScript disabled or blocked, charts won’t display.

Understanding Bot Behavior Patterns

Sometimes the data is accurate but confusing. Here are common patterns and what they mean:

Sudden drops in crawl activity might indicate:

• Your site was temporarily unavailable
• Changes to robots.txt blocking access
• Server performance issues causing timeouts
• Natural fluctuation in bot behavior (this is normal)

Seeing mostly homepage crawls suggests:

• Poor internal linking structure
• New content not being discovered
• Possible crawl budget issues on large sites

High number of 404 errors indicates:

• Deleted content still being requested
• Incorrect internal links
• Old external links pointing to moved content

Filter and Settings Not Saving

If your filter selections or settings aren’t persisting:

1. Ensure you’re clicking “Apply Filters” or “Save” buttons
2. Check that your browser isn’t blocking cookies
3. Verify you have proper user permissions (administrator access required)
4. Try logging out and back into WordPress

When to Contact Support

Reach out to support if you experience:

• Database errors appearing in the interface
• Complete feature failure after following troubleshooting steps
• Data that’s clearly incorrect (like timestamps from the future)
• PHP errors or white screens when accessing the feature

When contacting support, provide:

• Your WordPress version
• PHP version
• Current retention settings
• Screenshots of any error messages
• Description of what you expected vs. what you’re seeing

Best Practices for Reliable Data

Start with default settings and adjust gradually. The default configuration works well for most sites.

Allow time for data collection. It takes at least a week to see meaningful patterns and a month for reliable trends.

Focus on trends, not individual events. Bot behavior is naturally variable. Look for patterns over time rather than worrying about daily fluctuations.

Regular maintenance:

• Review your retention settings quarterly
• Clear very old data if performance degrades
• Update the plugin regularly for improved bot detection

Use the data strategically:

• Pages with declining crawl frequency might need content updates
• High error rates indicate technical SEO issues
• Coverage gaps show content discovery problems

Remember, crawler analysis is just one tool in your SEO toolkit. Use it alongside other metrics like search rankings, traffic, and user engagement for a complete picture of your site’s performance.