How to add additional bots (integrations maintained by Prerender)

Prerender provides a predefined list of allowed bots in its integrations. While it covers the majority of bots, you might wish to add additional bots to be served by Prerender.

Since every integration is different, the method of adding bots will also differ.

ExpressJS (Javascript) and NuxtJS
Rails(Ruby)
Nginx
Apache
IIS
Cloudflare
Docker

ExpressJS (Javascript) and NuxtJS

After you have downloaded the package via npm or yarn, open the index.js file in :

node_modules/prerender-node/index.js

Then add your desired bot user-agent name to the following list (the following list is not case-sensitive):

prerender.crawlerUserAgents = [
  'Google-InspectionTool',
  'googlebot',
  'Yahoo! Slurp',
  'bingbot',
  'yandex',
  'baiduspider',
  'facebookexternalhit',
  'twitterbot',
  'rogerbot',
  'linkedinbot',
  'embedly',
  'quora link preview',
  'showyoubot',
  'outbrain',
  'pinterest/0.',
  'developers.google.com/+/web/snippet',
  'slackbot',
  'vkShare',
  'W3C_Validator',
  'redditbot',
  'Applebot',
  'WhatsApp',
  'flipboard',
  'tumblr',
  'bitlybot',
  'SkypeUriPreview',
  'nuzzel',
  'Discordbot',
  'Google Page Speed',
  'Qwantify',
  'pinterestbot',
  'Bitrix link preview',
  'XING-contenttabreceiver',
  'Chrome-Lighthouse',
  'TelegramBot',
  'SeznamBot',
];

Rails(Ruby)

First, locate where prerender_rails is, with the following command :

gem which 'prerender_rails'

After opening the file, add your bot to the @crawler_user_agents array, then save it :

      @crawler_user_agents = [
        'Google-InspectionTool',
        'googlebot',
        'yahoo',
        'bingbot',
        'baiduspider',
        'facebookexternalhit',
        'twitterbot',
        'rogerbot',
        'linkedinbot',
        'embedly',
        'bufferbot',
        'quora link preview',
        'showyoubot',
        'outbrain',
        'pinterest/0.',
        'developers.google.com/+/web/snippet',
        'www.google.com/webmasters/tools/richsnippets',
        'slackbot',
        'vkShare',
        'W3C_Validator',
        'redditbot',
        'Applebot',
        'WhatsApp',
        'flipboard',
        'tumblr',
        'bitlybot',
        'SkypeUriPreview',
        'nuzzel',
        'Discordbot',
        'Google Page Speed',
        'Qwantify',
        'Chrome-Lighthouse',
        'TelegramBot'
      ]

Nginx

Look for $http_user_agent $prerender_ua and add the desired User Agent to the list.

    map $http_user_agent $prerender_ua {
        default       0;
        "~*Prerender" 0;

        "~*Google-InspectionTool"                   1;
        "~*googlebot"                               1;
        "~*yahoo!\ slurp"                           1;
        "~*bingbot"                                 1;
        "~*yandex"                                  1;
        "~*baiduspider"                             1;
        "~*facebookexternalhit"                     1;
        "~*twitterbot"                              1;
        "~*rogerbot"                                1;
        "~*linkedinbot"                             1;
        "~*embedly"                                 1;
        "~*quora\ link\ preview"                    1;
        "~*showyoubot"                              1;
        "~*outbrain"                                1;
        "~*pinterest\/0\."                          1;
        "~*developers.google.com\/\+\/web\/snippet" 1;
        "~*slackbot"                                1;
        "~*vkshare"                                 1;
        "~*w3c_validator"                           1;
        "~*redditbot"                               1;
        "~*applebot"                                1;
        "~*whatsapp"                                1;
        "~*flipboard"                               1;
        "~*tumblr"                                  1;
        "~*bitlybot"                                1;
        "~*skypeuripreview"                         1;
        "~*nuzzel"                                  1;
        "~*discordbot"                              1;
        "~*google\ page\ speed"                     1;
        "~*qwantify"                                1;
        "~*pinterestbot"                            1;
        "~*bitrix\ link\ preview"                   1;
        "~*xing-contenttabreceiver"                 1;
        "~*chrome-lighthouse"                       1;
        "~*telegrambot"                             1;
    }

Apache

In the .htaccess file, look for the line RewriteCond %{HTTP_USER_AGENT} , then add the User Agent to the list, separated by |

 RewriteCond %{HTTP_USER_AGENT} Google-InspectionTool|googlebot|bingbot|yandex|baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest\/0\.|pinterestbot|slackbot|vkShare|W3C_Validator|whatsapp|redditbot|applebot|flipboard|tumblr|bitlybot|skypeuripreview|nuzzel|discordbot|google\ page\ speed|qwantify|bitrix\ link\ preview|xing-contenttabreceiver|chrome-lighthouse|telegrambot [NC,OR]

IIS

Look for the {HTTP_USER_AGENT} condition, then add your desired new User Agent to the pattern, divide it with |

Google-InspectionTool|googlebot|bingbot|yandex|baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest\/0\.|pinterestbot|slackbot|vkShare|W3C_Validator|whatsapp|redditbot|applebot|flipboard|tumblr|bitlybot|skypeuripreview|nuzzel|discordbot|google\ page\ speed|qwantify|bitrix\ link\ preview|xing-contenttabreceiver|chrome-lighthouse|telegrambot

Cloudflare

Locate and edit the worker you made for Prerender and locate const BOT_AGENTS in the code, then add your desired User Agent to the end of the list

const BOT_AGENTS = [
  'google-inspectiontool'
  'googlebot',
  'yahoo! slurp',
  'bingbot',
  'yandex',
  'baiduspider',
  'facebookexternalhit',
  'twitterbot',
  'rogerbot',
  'linkedinbot',
  'embedly',
  'quora link preview',
  'showyoubot',
  'outbrain',
  'pinterest/0.',
  'developers.google.com/+/web/snippet',
  'slackbot',
  'vkshare',
  'w3c_validator',
  'redditbot',
  'applebot',
  'whatsapp',
  'flipboard',
  'tumblr',
  'bitlybot',
  'skypeuripreview',
  'nuzzel',
  'discordbot',
  'google page speed',
  'qwantify',
  'pinterestbot',
  'bitrix link preview',
  'xing-contenttabreceiver',
  'chrome-lighthouse',
  'telegrambot'
];

Docker

As our docker middleware uses Nginx, you will need to open the nginx.conf and locate map $http_user_agent $prerender_ua and add your desired User Agent to the list

map $http_user_agent $prerender_ua {
    default       0;
    "~*Prerender" 0;

    "~*google-inspectiontool"                   1;
    "~*googlebot"                               1;
    "~*yahoo!\ slurp"                           1;
    "~*bingbot"                                 1;
    "~*yandex"                                  1;
    "~*baiduspider"                             1;
    "~*facebookexternalhit"                     1;
    "~*twitterbot"                              1;
    "~*rogerbot"                                1;
    "~*linkedinbot"                             1;
    "~*embedly"                                 1;
    "~*quora\ link\ preview"                    1;
    "~*showyoubot"                              1;
    "~*outbrain"                                1;
    "~*pinterest\/0\."                          1;
    "~*developers.google.com\/\+\/web\/snippet" 1;
    "~*slackbot"                                1;
    "~*vkshare"                                 1;
    "~*w3c_validator"                           1;
    "~*redditbot"                               1;
    "~*applebot"                                1;
    "~*whatsapp"                                1;
    "~*flipboard"                               1;
    "~*tumblr"                                  1;
    "~*bitlybot"                                1;
    "~*skypeuripreview"                         1;
    "~*nuzzel"                                  1;
    "~*discordbot"                              1;
    "~*google\ page\ speed"                     1;
    "~*qwantify"                                1;
    "~*pinterestbot"                            1;
    "~*bitrix\ link\ preview"                   1;
    "~*xing-contenttabreceiver"                 1;
    "~*chrome-lighthouse"                       1;
    "~*telegrambot"                             1;
}