Web Crawler and Bot Reference Guide

Understanding which bots visit your site is key to interpreting your crawl-log data. These automated programs serve very different purposes — search engine indexing, AI training, social media previews, SEO research.

2025 reality check: AI crawlers now make up over 50% of web traffic on many sites. Some collect data for AI training (you may want to block them); others fetch content when users ask AI assistants questions (these can drive referral traffic to your site).

How Linkilo uses this list

Linkilo recognises each bot below from the text in its user-agent string (the Detection pattern lines). Every recognised bot gets its own line in the Crawler Analyzer charts; anything that calls itself a bot, crawler or spider but is not on the list is grouped under Unrecognized Bots/Crawlers.

You choose which bots are recorded under Linkilo → Settings → Server Log Analyzer. Turn on Record crawler visits, pick how long to keep history under Keep visit history for, then tick bots individually or use Select all, Deselect all, Search engines only or AI crawlers only. Not every bot is recorded on a fresh install — AhrefsBot and SemrushBot, for example, are off by default because they are noisy. Click Save Settings when done.

Search engine crawlers

These index your pages so they appear in search results. Blocking them removes you from that search engine.

Googlebot

• Operator: Google
• Detection pattern: googlebot, google-structured-data, storebot-google, google-inspectiontool, adsbot-google
• Purpose: Google's primary crawler. Operates in Desktop and Smartphone versions for mobile-first indexing.
• Why track: Directly impacts your Google Search visibility. Higher frequency means Google sees your content as important.
• Recommendation: Always Allow

Bingbot

• Operator: Microsoft
• Detection pattern: bingbot, msnbot, bingpreview
• Purpose: Crawls for Bing search. Also powers Copilot and Microsoft Edge suggestions.
• Why track: Essential for Bing users, Edge, Cortana, and Office integration.
• Recommendation: Always Allow

DuckDuckBot

• Operator: DuckDuckGo
• Detection pattern: duckduckbot, duckduckgo
• Purpose: Crawls for DuckDuckGo's privacy-focused search.
• Recommendation: Recommended

YandexBot

• Operator: Yandex
• Detection pattern: yandexbot, yandex
• Purpose: Russia's leading search engine crawler.
• Recommendation: Regional (allow if targeting Russian-speaking markets)

Baiduspider

• Operator: Baidu
• Detection pattern: baiduspider, baidu
• Purpose: China's dominant search crawler.
• Recommendation: Regional (allow if targeting China)

Yahoo Slurp

• Operator: Yahoo
• Detection pattern: slurp
• Purpose: Historically Yahoo's primary crawler; Yahoo now largely uses Bing's index.
• Recommendation: Optional

Applebot

• Operator: Apple
• Detection pattern: applebot
• Purpose: Powers Siri suggestions, Spotlight, Safari intelligent features.
• Recommendation: Recommended

Sogou Spider

• Operator: Sogou (Tencent)
• Detection pattern: sogou
• Purpose: Chinese search engine alternative to Baidu, integrated with WeChat.
• Recommendation: Regional (China)

PetalBot

• Operator: Huawei
• Detection pattern: petalbot
• Purpose: Powers Petal Search on Huawei devices without Google services.
• Recommendation: Regional

AI training crawlers

These collect content to train AI/LLM models. Blocking them will NOT affect your search rankings.

Safe to block: you can block every AI training bot without any impact on Google, Bing, or other search engine visibility.

GPTBot

• Operator: OpenAI
• Detection pattern: gptbot
• Purpose: Collects training data for GPT models (ChatGPT, GPT-4). Highest-volume AI crawler — ~30% of all AI bot traffic.
• Recommendation: Your Choice (safe to block)

Google-Extended

• Operator: Google
• Detection pattern: google-extended
• Purpose: Training data for Google's Gemini. Completely separate from Googlebot.
• Recommendation: Your Choice (safe to block — does NOT affect Google Search)

ClaudeBot

• Operator: Anthropic
• Detection pattern: claudebot, claude-web
• Purpose: Training data for Claude AI. Respects robots.txt.
• Recommendation: Your Choice (safe to block)

Anthropic-AI

• Operator: Anthropic
• Detection pattern: anthropic-ai
• Purpose: Older Anthropic crawler identifier, tracked separately from ClaudeBot.
• Recommendation: Your Choice (safe to block)

Meta-ExternalAgent

• Operator: Meta
• Detection pattern: meta-externalagent, meta-externalfetcher
• Purpose: Training data for Llama models. ~19% of AI crawler traffic. Different from Facebook's social preview bot.
• Recommendation: Your Choice (safe to block)

PerplexityBot

• Operator: Perplexity AI
• Detection pattern: perplexitybot
• Purpose: Fastest-growing AI crawler — 157,000%+ growth in 2024.
• Recommendation: Your Choice (safe to block)

CCBot

• Operator: Common Crawl Foundation
• Detection pattern: ccbot
• Purpose: Non-profit. Data ends up in many AI systems indirectly.
• Recommendation: Your Choice (safe to block)

Bytespider

• Operator: ByteDance (TikTok)
• Detection pattern: bytespider
• Purpose: ByteDance's AI crawler. Currently declining in volume.
• Recommendation: Your Choice (safe to block)

Grok

• Operator: xAI
• Detection pattern: grok, xai
• Purpose: Powers Grok with real-time web access; integrated with X (Twitter).
• Recommendation: Your Choice (safe to block)

Cohere-AI

• Operator: Cohere
• Detection pattern: cohere-ai
• Purpose: Enterprise AI training data.
• Recommendation: Your Choice (safe to block)

AI2Bot

• Operator: Allen Institute for AI
• Detection pattern: ai2bot
• Purpose: Academic AI research data.
• Recommendation: Your Choice (safe to block)

Diffbot

• Operator: Diffbot
• Detection pattern: diffbot
• Purpose: AI-powered web scraping and knowledge graph building.
• Recommendation: Your Choice (safe to block)

Applebot-Extended

• Operator: Apple
• Detection pattern: applebot-extended
• Purpose: Apple Intelligence training. Separate from regular Applebot.
• Recommendation: Your Choice (safe to block — does NOT affect Siri/Spotlight)

Amazonbot

• Operator: Amazon
• Detection pattern: amazonbot
• Purpose: Powers Alexa answers and AI training.
• Recommendation: Your Choice (blocking may affect how Alexa answers about your content)

YouBot

• Operator: You.com
• Detection pattern: youbot
• Recommendation: Your Choice (safe to block)

Omgilibot

• Operator: Omgili / Webz.io
• Detection pattern: omgili
• Purpose: Powers news and social media monitoring.
• Recommendation: Your Choice (safe to block)

AI assistant crawlers (user-triggered)

These fetch pages when a real user asks an AI assistant a question. They can drive referral traffic when AI cites your content.

Consider allowing these. Unlike training crawlers, these represent actual users seeking information. When AI cites your content, those users may click through to your site.

ChatGPT-User

• Operator: OpenAI
• Detection pattern: chatgpt-user
• Purpose: Real-time fetch when ChatGPT users ask questions. 2,825%+ growth in 2024.
• Recommendation: Consider Allowing (can drive traffic)

OAI-SearchBot

• Operator: OpenAI
• Detection pattern: oai-searchbot
• Purpose: Powers ChatGPT's search feature.
• Recommendation: Consider Allowing

Claude-User

• Operator: Anthropic
• Detection pattern: claude-user
• Purpose: Fetches content when Claude users request current information.
• Recommendation: Consider Allowing

Perplexity-User

• Operator: Perplexity AI
• Detection pattern: perplexity-user
• Purpose: Powers Perplexity's AI search answers. Note: known to sometimes ignore robots.txt.
• Recommendation: Consider Allowing (high referral potential)

MistralAI-User

• Operator: Mistral AI
• Detection pattern: mistralai-user
• Purpose: Fetches pages when Le Chat users ask for current information.
• Recommendation: Consider Allowing

DuckAssistBot

• Operator: DuckDuckGo
• Detection pattern: duckassistbot
• Purpose: Powers DuckDuckGo AI answers. Tracked separately from the regular DuckDuckBot search crawler.
• Recommendation: Consider Allowing

Social media crawlers

These generate link previews when your URLs are shared. Blocking them breaks your link previews.

facebookexternalhit

• Operator: Meta
• Detection pattern: facebookexternalhit, facebot
• Purpose: Link previews for Facebook, Instagram, Messenger, WhatsApp.
• Recommendation: Always Allow

Twitterbot

• Operator: X (Twitter)
• Detection pattern: twitterbot
• Recommendation: Always Allow

LinkedInBot

• Operator: LinkedIn
• Detection pattern: linkedinbot
• Recommendation: Always Allow

Pinterest

• Operator: Pinterest
• Detection pattern: pinterest, pinterestbot
• Purpose: Indexes images for Pinterest's visual discovery.
• Recommendation: Always Allow

Slackbot

• Operator: Slack (Salesforce)
• Detection pattern: slackbot
• Recommendation: Recommended

Discordbot

• Operator: Discord
• Detection pattern: discordbot
• Recommendation: Recommended

TelegramBot

• Operator: Telegram
• Detection pattern: telegrambot
• Recommendation: Recommended

WhatsApp

• Operator: Meta
• Detection pattern: whatsapp
• Recommendation: Always Allow

SEO tool crawlers

Commercial bots gathering data for SEO analysis tools. Generally safe to allow but can be resource-intensive.

AhrefsBot

• Operator: Ahrefs
• Detection pattern: ahrefsbot
• Purpose: Backlink database, keyword research, site audits.
• Recommendation: Allow (consider rate-limiting if needed). Not recorded by default in Linkilo.

SemrushBot

• Operator: Semrush
• Detection pattern: semrushbot
• Recommendation: Allow (consider rate-limiting if needed). Not recorded by default in Linkilo.

MJ12bot

• Operator: Majestic
• Detection pattern: mj12bot
• Purpose: Trust Flow, Citation Flow, link authority metrics.
• Recommendation: Allow

DotBot

• Operator: Moz
• Detection pattern: dotbot
• Purpose: Calculates Domain Authority and Page Authority.
• Recommendation: Allow

Rogerbot

• Operator: Moz
• Detection pattern: rogerbot
• Recommendation: Allow

Screaming Frog

• Operator: Screaming Frog Ltd
• Detection pattern: screaming frog, seospider
• Purpose: Popular desktop SEO crawler — often used by SEO pros (yours or competitors').
• Recommendation: Allow

SEOkicks

• Operator: SEOkicks
• Detection pattern: seokicks
• Recommendation: Allow

SISTRIX

• Operator: SISTRIX
• Detection pattern: sistrix
• Purpose: Powers SISTRIX visibility index — popular in European markets.
• Recommendation: Allow

SerpstatBot

• Operator: Serpstat
• Detection pattern: serpstatbot
• Recommendation: Allow

Unrecognized Bots/Crawlers

Any visitor whose user-agent contains bot, crawl, spider, slurp or fetcher but matches none of the patterns above lands in the Unrecognized Bots/Crawlers line (called “Other Bots” in older versions). A sudden jump here usually means a new scraper; open the Crawler Analyzer and check the user-agents behind it.

Quick robots.txt reference

Block all AI training bots

User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: PerplexityBot
Disallow: /

Allow AI assistant bots (for referral traffic)

User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: OAI-SearchBot
Allow: /

Not all bots respect robots.txt. For stronger control, use server-level blocking via .htaccess rules or a Web Application Firewall (WAF).

---

This reference is part of Linkilo's Crawler Analyzer feature.

Last reviewed for Linkilo 3.0.0 (October 2026).