Recommended AI Crawler User Agents for Prerender.io
Allowing AI crawlers to access your website and Prerender.io is essential to ensure your content appears in AI-generated search results and summaries.
Allowing these bots helps your site appear in:
-
ChatGPT answers
-
Claude (Anthropic) summaries
-
Perplexity search responses
-
Google AI Overviews and Gemini
-
Bing AI, You.com, and other LLM tools
For instance, one customer received over 700,000 requests from ChatGPT in just 30 days after adding the GPTBot user agent.
Open our How to add additional bots article for detailed instructions on how to configure your site to send requests from AI crawlers (and other bots) to Prerender.
⚙️ Developers may need to assist with implementation depending on your setup.
Major AI Search Crawlers
-
openai/chatgpt -
chatgpt-user -
gptbot -
chatgpt-user/1.0 -
anthropic/claude -
anthropic-ai -
anthropic/claude -
google-extended -
googleother -
microsoft/bing ai -
bingbot/ai -
bingpreview -
perplexity -
perplexitybot -
perplexitybot/1.0 -
cohere -
cohere-ai -
cohere-crawler
Emerging AI Platforms
-
bytedance/tiktok -
bytespider -
bytespiderbot -
baiduspider-render -
baiduspider-ai -
yourbot (you.com) -
you-bot -
neevabot -
hugging-face-ai -
huggingfacebot
AI-Powered Social & Content Platforms
-
linkedinbot/ai -
facebookexternalhit/ai -
facebookbot -
twitterbot -
redditbot
AI Research & Specialized Crawlers
-
ccbot (common crawl) -
airesearchbot -
research-crawler -
diffbot -
scaleai -
scale-crawler
Need Help?
If you're unsure how to configure these settings, contact our support team or message us via your Prerender dashboard — we’re happy to assist!