How do I add AI crawlers and bots to my integration?

TL;DR

Prerender.io use user-agent string matching to decide which requests are forwarded using a list of known bot user-agent strings. If a crawler you want to support isn't on the default list, find its user-agent string, then add it to the relevant config for your integration type: Express, Rails, Nginx, Apache, IIS, or Cloudflare. Changes take effect after a server restart or redeploy.

---

Why the default list may not be enough

Prerender.io integrations ship with a predefined set of known AI crawler and search engine user agents. This covers the majority of traffic from GPTBot, ClaudeBot, Googlebot, BingBot, and others.

You may need to extend that list if you want to serve prerendered content to less common crawlers like applebot or duckduckbot, AI platforms that have released new crawlers since your integration was last updated, or internal QA tools that simulate crawler requests.

➡️ See the full list of bots Prerender.io can recognize and classify to check whether the bot you need is already covered.

 

---

Step 1: Find the user-agent string for the bot you want to add.

Each crawler identifies itself with a user-agent string. For example, Googlebot uses googlebot and the Apple crawler uses applebot. You need the exact string (or a reliable substring) to match it in your config.

Check the crawler's official documentation for the correct user-agent value. A case-insensitive substring match is enough for most integrations — you do not need to match the full string.

---

Step 2: Add the user-agent string to your integration.

The method depends on your integration type. Find your platform below.

Express and Nuxt.js

To append bots without replacing the default list (recommended for prerender-node v3.8.2 or higher):

js

app.use(   require('prerender-node')     .set('prerenderToken', 'YOUR_TOKEN')     .addUserAgents(['your-custom-bot'])     .addUserAgents('another-custom-bot') );

addUserAgents accepts an array or a single string. Duplicates are ignored.

To replace the entire default list (use with caution—you must include every bot you want to serve): 

js

app.use(   require('prerender-node')     .set('prerenderToken', 'YOUR_TOKEN')     .set('crawlerUserAgents', ['googlebot', 'your-custom-bot']) );

To view the current default crawler list, open:

node_modules/prerender-node/index.js

Look for the prerender.crawlerUserAgents array. You can also view it in this reference Gist.

 

Rails (Ruby)

Instead, configure custom user agents in a Rails initializer. Create or open config/initializers/prerender_rails.rb and add: 

ruby

PrerenderRails.configure do |config|   config.crawler_user_agents << 'your-custom-bot' end

If a config option is not available, locate the gem with:

bash

gem which 'prerender_rails'

Then review the @crawler_user_agents array — but plan to move to a proper override as soon as the gem supports one.

 

Nginx

Locate the map $http_user_agent $prerender_ua block in your Nginx config and add a new line for your bot:

nginx

map $http_user_agent $prerender_ua {     default       0;     "~*Prerender" 0;      "~*google-inspectiontool"   1;     "~*your-custom-bot"         1; }

The ~* prefix makes matching case-insensitive. Restart Nginx after saving.

 

Apache

Open your .htaccess file, find the RewriteCond %{HTTP_USER_AGENT} line, and append your bot to the end of the pattern separated by |:

RewriteCond %{HTTP_USER_AGENT} googlebot|your-custom-bot [NC,OR]

 

IIS

Find the {HTTP_USER_AGENT} condition in your IIS config and add your bot to the pattern, separated by |:

googlebot|bingbot|your-custom-bot

 

Cloudflare

Open the Cloudflare Worker you created for Prerender.io and locate the BOT_AGENTS array. Add your bot to the end of the list:

js

const BOT_AGENTS = [   "googlebot",   "your-custom-bot", ];

 

Docker

The Prerender.io Docker integration uses Nginx internally. Open nginx.conf, locate map $http_user_agent $prerender_ua, and add your bot:

map $http_user_agent $prerender_ua {     default       0;     "~*Prerender" 0;      "~*googlebot"         1;     "~*your-custom-bot"   1; }

Restart the container after saving.

---

Step 3: Verify that your integration is serving the new bot.

After making your change and restarting or redeploying, confirm the new bot is receiving prerendered content.

➡️ See Verify your integration for full instructions.

For a quick check, run curl from your terminal using the bot's user-agent string:

curl -A "your-custom-bot" https://www.yoursite.com/

---

---

Related articles

• How do I verify my Prerender.io integration is working?
• Full list of bots Prerender.io can recognize and classify
• How to control crawling and indexing
• How do I add Googlebot to my Prerender.io integration?