Recommended AI Crawler User Agents for Prerender
As AI-powered search engines grow in popularity, it's crucial to ensure your site is accessible to their crawlers. Many of these engines like ChatGPT, Claude, Perplexity, and Google's Gemini use specialized bots to discover and index content. Since most can't execute JavaScript, allowing their crawlers through Prerender is essential to ensure your content appears in AI-generated search results and summaries.
Why Allow AI Crawlers?
Allowing these bots helps your site appear in:
-
ChatGPT answers
-
Claude (Anthropic) summaries
-
Perplexity search responses
-
Google AI Overviews and Gemini
-
Bing AI, You.com, and other LLM tools
For instance, one customer received over 700,000 requests from ChatGPT in just 30 days after adding the GPTBot user agent.
To ensure AI crawlers can access your content follow our guide.
⚙️ Developers may need to assist with implementation depending on your setup.
Recommended AI Crawler User Agents:
Major AI Search Crawlers
-
openai/chatgpt -
chatgpt-user -
gptbot -
chatgpt-user/1.0 -
anthropic/claude -
anthropic-ai -
anthropic/claude -
google-extended -
googleother -
microsoft/bing ai -
bingbot/ai -
bingpreview -
perplexity -
perplexitybot -
perplexitybot/1.0 -
cohere -
cohere-ai -
cohere-crawler
Emerging AI Platforms
-
bytedance/tiktok -
bytespider -
bytespiderbot -
baiduspider-render -
baiduspider-ai -
yourbot (you.com) -
you-bot -
neevabot -
hugging-face-ai -
huggingfacebot
AI-Powered Social & Content Platforms
-
linkedinbot/ai -
facebookexternalhit/ai -
facebookbot -
twitterbot -
redditbot
AI Research & Specialized Crawlers
-
ccbot (common crawl) -
airesearchbot -
research-crawler -
diffbot -
scaleai -
scale-crawler
Need Help?
If you're unsure how to configure these settings, contact our support team or message us via your Prerender dashboard — we’re happy to assist!