List of bots Prerender.io can recognize and classify
Every AI crawler, search engine, social media bot, and SEO tool that Prerender.io identifies and tracks in your dashboard.
TL;DR
Prerender.io serves pre-rendered HTML to every bot that requests it, including bots not in this list. Recognized bots are grouped by category in your dashboard. Unrecognized bots appear under Other. If a specific bot is not receiving your content, see how to add additional bots.
Prerender.io delivers your pages to all bots that request them, including those not listed below. In your dashboard, requests from unrecognized bots are grouped under Other.
AI crawlers
| Name | User agent(s) | Description |
|---|---|---|
| ChatGPT | gptbot, oai-searchbot, chatgpt-user |
OpenAI's crawlers that index web content for training AI models, powering search features, and browsing capabilities in ChatGPT. |
| Claude | claudebot, claude-user, claude-searchbot |
Anthropic's crawlers. claudebot collects training data; claude-user handles user-initiated browsing; claude-searchbot indexes content for Claude search features. |
| Gemini | google-extended |
Google's robots.txt opt-out token for Gemini and Vertex AI training data. Note: google-extended is a robots.txt signal, not a standalone crawler — Google uses its standard user agents for actual crawling. |
| Perplexity AI | perplexitybot, perplexity-user |
Indexes web content for Perplexity's AI-powered search engine and fetches real-time information for user queries. |
| You.com | youbot |
You.com's web crawler that powers its AI-driven search assistant. |
| Amazon | amazonbot |
Amazon's general-purpose AI training crawler used to improve Amazon's products and services. |
Search engines
| Name | User agent(s) | Description |
|---|---|---|
googlebot, adsbot-google, apis-google, mediapartners-google, google-safety, feedfetcher-google, googleproducer, google-site-verification |
Multiple Google user agents. googlebot indexes pages for Search. adsbot-google audits landing pages for Google Ads. mediapartners-google serves personalized AdSense ads. googleproducer fetches feeds for Google Publisher Center / Google News. google-site-verification is a one-time ownership verification fetcher for Search Console. |
|
| Bing | bingbot |
Microsoft's crawler that indexes content for Bing search results. |
| Yandex | yandexbot |
Indexes pages for Yandex search. (yabrowser is the end-user Yandex browser and appears in logs as human traffic, not bot traffic.) |
| Yahoo | yahoo |
Web crawler for Yahoo search and content indexing. |
| Baidu | baiduspider |
Indexes websites for Baidu search, China's leading search engine. |
| Naver | naver, yeti |
Indexes websites for Naver search, South Korea's leading search engine. Naver's crawler officially identifies as Yeti. |
| Seznam | seznambot, sznprohlizec |
Indexes websites for Seznam search, Czech Republic's leading search engine. |
| Qwant | qwantbot |
Privacy-focused web crawler used by Qwant to index public web pages. |
| Ecosia | ecosia |
Matches the Ecosia browser app. Ecosia does not operate its own indexing crawler — its search results come from Bing. This token appears when users browse with the Ecosia browser. |
| DuckDuckGo | duckduckbot, duckassistbot |
duckduckbot indexes websites for DuckDuckGo search. duckassistbot collects data for DuckDuckGo's AI-powered features and assistant. |
| Apple | applebot |
Indexes content for Siri, Spotlight suggestions, Safari search suggestions, and other Apple services. |
Social media
| Name | User agent(s) | Description |
|---|---|---|
facebookexternalhit, facebookcatalog, facebookbot, meta-externalagent |
Meta's crawlers that fetch link previews, index product catalogs, gather web content, and support features across Meta's apps. | |
| Twitter / X | twitterbot |
Crawls shared URLs to generate Twitter Cards and link previews. |
linkedinbot |
Fetches shared URLs to display link previews on LinkedIn posts. | |
whatsapp |
Retrieves preview data for links shared in WhatsApp chats. | |
| Slack | slackbot |
Requests webpage data to generate link previews in Slack messages. |
pinterestbot |
Indexes content and metadata for Pins and shared URLs. | |
| TikTok / ByteDance | tiktokspider, bytespider |
tiktokspider fetches webpage metadata for TikTok link previews. bytespider is ByteDance's general AI training crawler. |
| Discord | discordbot |
Generates previews for URLs shared in Discord channels. |
SEO tools
| Name | User agent(s) | Description |
|---|---|---|
| SEMrush | semrushbot |
Crawls websites to analyze SEO performance and backlink profiles. |
| Ahrefs | ahrefsbot |
Collects data on backlinks, rankings, and SEO metrics for analysis. |
| Chrome Lighthouse | chrome-lighthouse |
Google's automated auditing tool for web performance, accessibility, and SEO. This token matches local Lighthouse runs; the Google PageSpeed Insights API may use a different user agent. |
| Screaming Frog | screaming-frog |
Desktop-based crawler that audits website structure and metadata. Note: the actual user agent sent is Screaming Frog SEO Spider/x.x — screaming-frog is Prerender.io's matching alias. |
| OnCrawl | oncrawlbot |
Enterprise crawler used for site analysis and crawl performance insights. |
| Botify | botifybot |
Analyzes site architecture and crawling efficiency. |
| Lumar | deepcrawl, lumar |
Technical SEO crawlers that analyze and report on site health and indexability. |
| Moz | rogerbot, dotbot |
rogerbot triggers Moz Pro site audits for individual customers. dotbot is Moz's general web crawler that builds the link index. |
A bot isn't receiving your content?
If your website isn't delivering content to a specific bot, see how to add additional bots for instructions.
Related articles
- How to add additional bots
- How Prerender.io works
- Verify your Prerender.io integration
- Available integrations
Still need help? If you're seeing unexpected bot behavior or a bot you need isn't listed here, our support team can help. → Contact us at support@prerender.io