List of bots Prerender.io can recognize and classify
This article includes a comprehensive list of bots that Prerender.io can track and display on your Dashboard.
Important: Prerender will deliver your pages to all bots that request them, including those not listed below. In the Dashboard, those requests/bots will be grouped under the "Other" section.
If your website isn't delivering content to some of these bots please open this article for instructions on how to change that.
Search engines
| Name | User agent(s) | Description |
|
googlebot, adsbot-google, apis-google, mediapartners-google, google-safety, feedfetcher-google, googleproducer, google-site-verification
|
Crawls, indexes, verifies, and analyses web content for various Google services.
|
|
| Bing |
bingbot
|
Microsoft's crawler that indexes content for Bing's search results.
|
| Yandex |
yandexbot, yabrowser
|
Indexes pages for search, and their web browser allows end users to access websites. |
| Yahoo |
yahoo
|
Web crawler for Yahoo search and content indexing.
|
| Baidu |
baiduspider
|
Indexes websites for Baidu search, China's leading search engine. |
| Naver |
naver
|
Indexes websites for Naver search, South Korea's leading search engine.
|
| Seznam |
seznambot,sznprohlizec
|
Indexes websites for Seznam search, Czech Republic's leading search engine.
|
| Qwant |
qwantbot
|
Privacy-focused web crawler used by Qwant to index public web pages.
|
| Ecosia |
ecosia
|
Indexes websites for Ecosia's eco-friendly search engine.
|
| DuckDuckGo |
duckduckbot,duckassistbot
|
duckduckbot - Indexes websites and for their search engine. duckassistbot - Collects data for their AI-powered features and assistant functionalities. |
| Apple |
applebot
|
Indexes content for Siri, Spotlight suggestions, Safari search suggestions, and other Apple services.
|
Social media
| Name | User agent(s) | Description |
|
facebookexternalhit,facebookcatalog,facebookbot,meta-externalagent
|
Meta's crawlers that fetch link previews, index product catalogs, gather web content, and support features across Meta's apps.
|
|
| Crawls external links to generate preview cards and metadata for URLs shared on Instagram. |
||
| twitterbot | Crawls shared URLs to generate Twitter Cards and previews. | |
| linkedinbot | Fetches shared URLs to display link previews on LinkedIn posts. | |
| Retrieves preview data for links shared in WhatsApp chats. | ||
| Slack | slackbot | Requests webpage data to generate link previews in Slack messages |
| Indexes content and metadata for Pins and shared URLs. | ||
| TikTok | tiktok,tiktokspider | Fetches webpage metadata for TikTok link previews. |
| Discord | discordbot | Generates previews for URLs shared in Discord channels. |
SEO tools
| Name | User agent(s) | Description |
| SEMRush |
semrushbot
|
Crawls websites to analyse SEO performance and backlink profiles.
|
| Ahrefs | ahrefsbot | Collects data on backlinks, rankings, and SEO metrics for analysis. |
| Chrome Lighthouse |
chrome-lighthouse
|
Google’s automated auditing tool for web performance, accessibility, and SEO.
|
| Screaming Frog | screaming-frog | Desktop-based crawler that audits website structure and metadata. |
| OnCrawl |
oncrawlbot
|
Enterprise crawler used for site analysis and crawl performance insights.
|
| Botify | botifybot | Analyses architecture and crawling efficiency. |
| Lumar |
deepcrawl, lumar
|
Technical SEO crawlers that analyse and report on site health and indexability.
|
| Moz |
rogerbot, dotbot
|
Collect backlink and keyword ranking data.
|
AI
| Name | User agent(s) | Description |
| ChatGPT | gptbot, chatgpt,oai-searchbot,chatgpt-user | OpenAI's crawlers that index web content for training AI models, powering search features, and browsing capabilities in ChatGPT. |
| Claude | claudebot | Anthropic’s crawler used to collect web data for Claude’s AI models and research |
| Gemini |
google-extended
|
Google’s AI data crawler for training and enhancing Gemini and Vertex AI products
|
| Perplexity AI |
perplexitybot,perplexity-user
|
Indexes web content for their AI-powered search engine and fetch real-time information for user queries.
|
| You.com | youbot | You.com’s web crawler that powers its AI-driven search assistant |
| Amazon | amazonbot | Amazon’s AI crawler used to used to improve their services, such as enabling Alexa to more accurately answer questions for customers |