Skip to content
Contact us
  • There are no suggestions because the search field is empty.

Recommended AI Crawler User Agents for Prerender.io

A complete reference list of AI crawler user agents — route them through Prerender to maximize your content's reach in AI-generated answers and search results.

TL;DR

AI crawlers identify themselves by a user agent string — Prerender routes requests to your pre-rendered pages based on which user agents you configure. This article lists the AI crawler user agents worth including, organized by platform type. To configure which user agents Prerender routes, see How to add additional bots.

When an AI crawler visits your site, it sends a user agent string identifying which platform sent it — GPTBot for OpenAI, ClaudeBot for Anthropic, PerplexityBot for Perplexity, and so on. Prerender uses these strings to identify crawler requests and serve pre-rendered HTML instead of a JavaScript shell that many crawlers can't read.

Configure the right user agents and your content becomes eligible for inclusion in AI-generated answers from ChatGPT, citations from Claude, Perplexity search responses, Google AI Overviews, and Bing AI results. Miss a user agent and that platform's crawler may receive an incomplete page — or skip your content entirely.

One customer saw over 700,000 requests from GPTBot alone in a single 30-day period after configuring the user agent. AI crawler traffic can be substantial once your content is reachable. See the integration setup guide if you haven't connected Prerender to your stack yet.

ℹ️ To configure which user agents Prerender routes, follow the steps in How to add additional bots. Depending on your setup, a developer may need to assist with implementation.

Major AI search crawlers

These user agents come from the primary AI platforms generating the most traffic from AI citations and search integrations. Prioritize these first.

  • openai/chatgpt
  • chatgpt-user
  • gptbot
  • chatgpt-user/1.0
  • anthropic/claude
  • anthropic-ai
  • google-extended
  • googleother
  • microsoft/bing ai
  • bingbot/ai
  • bingpreview
  • perplexity
  • perplexitybot
  • perplexitybot/1.0
  • cohere
  • cohere-ai
  • cohere-crawler

Emerging AI platforms

These crawlers come from platforms growing in AI search traffic. Including them ensures your content is reachable as these platforms expand their reach.

  • bytedance/tiktok
  • bytespider
  • bytespiderbot
  • baiduspider-render
  • baiduspider-ai
  • yourbot (you.com)
  • you-bot
  • neevabot
  • hugging-face-ai
  • huggingfacebot

AI-powered social & content platforms

These crawlers power social sharing previews and AI-assisted content features on major social networks.

  • linkedinbot/ai
  • facebookexternalhit/ai
  • facebookbot
  • twitterbot
  • redditbot

AI research & specialized crawlers

These crawlers collect data for AI model training and power specialized AI tools. Including them extends your content's reach into research-driven and data-intensive AI applications.

  • ccbot (common crawl)
  • airesearchbot
  • research-crawler
  • diffbot
  • scaleai
  • scale-crawler

💬 Still need help?
If you're unsure which user agents to prioritize or how to configure them for your setup, our support team can help.
→ Contact us at support@prerender.io

Related articles

Was this article helpful?