How do I add AI crawlers and bots to my integration?

TL;DR

Prerender.io uses User-Agent matching to decide which requests to pre-render. To serve a bot that isn't on the default list, find the bot list in your integration configuration and add the bot's User-Agent string. The exact location differs by platform: Express.js, Rails, Nginx, Apache, IIS, Cloudflare, and Docker each have their own configuration file or method.

---

Why add additional bots

Prerender.io ships with a default list of known AI crawlers and search engines. New crawlers appear regularly, and some may not be on the default list for your integration version.

Common reasons to add a bot:

• AI crawlers. GPTBot, ClaudeBot, PerplexityBot, and others may not appear in older integration versions.
• Internal or QA tools. Custom crawlers used for testing or content previews need pre-rendered HTML to work correctly.
• Specialist SEO tools. Some SEO platforms simulate crawlers that don't match the default User-Agent patterns.

---

Step 1: find the user-agent string for the bot you want to add.

Each crawler identifies itself with a user-agent string. For example, Googlebot uses googlebot and the Apple crawler uses applebot. You need the exact string (or a reliable substring) to match it in your config.

Check the crawler's official documentation for the correct user-agent value. A case-insensitive substring match is enough for most integrations — you do not need to match the full string.

---

Step 2: add the User-Agent to your integration.

The location of the bot list depends on which integration you use. Find your platform below.

Express.js and Nuxt.js

The prerender-node package exposes two methods for adding User-Agent strings.

Recommended: append to the default list (prerender-node versions after v3.8.2)

Use addUserAgents to add strings without replacing the default list. It accepts a single string or an array. Duplicates are ignored automatically.

app.use(  
  require('prerender-node')  
    .set('prerenderToken', 'YOUR_TOKEN')  
    .addUserAgents(['your-custom-bot'])  
    .addUserAgents('another-bot') 
);

Replace the entire list (any version)

Use the set method with crawlerUserAgents. This replaces the default list entirely.

app.use( 
 require('prerender-node') 
 .set('prerenderToken', 'YOUR_TOKEN') 
 .addUserAgents(['your-custom-bot']) 
 .addUserAgents('another-bot') 
);

To view the current default list, open this file in your project:

node_modules/prerender-node/index.js

Look for the prerender.crawlerUserAgents array. You can also view the default list on GitHub.

---

Rails (Ruby)

Find the prerender_rails gem file with:

gem which 'prerender_rails'

Open the file and add your bot's User-Agent string to the @crawler_user_agents array, then save: 

@crawler_user_agents = [  
   'google-inspectiontool',  
   'your-custom-bot',  ... 
]

Restart your Rails server for the change to take effect.

---

Nginx

Locate the map $http_user_agent $prerender_ua block in your Nginx config and add a new line for your bot:

nginx

map $http_user_agent $prerender_ua {     
     default       0;     
     "~*Prerender" 0;      
     "~*google-inspectiontool"   1;     
     "~*your-custom-bot"         1;
     ... 
}

The ~* prefix makes matching case-insensitive. Restart Nginx after saving.

---

Apache

Open your .htaccess file, find the RewriteCond %{HTTP_USER_AGENT} line, and append your bot to the end of the pattern separated by |:

RewriteCond %{HTTP_USER_AGENT} google-inspectionTool|googlebot|your-custom-bot [NC,OR]

---

IIS

Find the {HTTP_USER_AGENT} condition in your IIS config and add your bot to the pattern, separated by |:

google-inspectionTool|googlebot|bingbot|your-custom-bot

---

Cloudflare

Open the Cloudflare Worker you created for Prerender.io and locate the BOT_AGENTS array. Add your bot to the end of the list:

const BOT_AGENTS = [   
   "googlebot",   
   "your-custom-bot", 
   ... 
];

---

Docker

The Prerender.io Docker integration uses Nginx internally. Open nginx.conf, locate map $http_user_agent $prerender_ua, and add your bot:

map $http_user_agent $prerender_ua {     
    default       0;     
    "~*Prerender" 0;      
    "~*googlebot" 1;     
    "~*your-custom-bot" 1; 
    ... 
}

Restart the Docker container after saving.

---

Step 3: verify that your integration is serving the new bot.

After making your change and restarting or redeploying, confirm the new bot is receiving prerendered content.

For a quick check, run curl from your terminal using the bot's user-agent string:

curl -A "your-custom-bot" https://www.yoursite.com/

If the response contains fully rendered HTML rather than a JavaScript shell, the bot is configured correctly. 

---

---

Related articles

• How do I verify my Prerender.io integration is working?
• List of bots Prerender.io can recognize and classify
• How to control crawling and indexing
• How do I add Googlebot to my Prerender.io integration?
  
  

---