Adding Googlebot to your integration

 

Overview

Google recommends using Prerender.io for Dynamic Rendering to serve prerendered content to search engines and JavaScript content to users. (Dynamic Rendering documentation)

In May 2018, Google introduced Dynamic Rendering as a strategy for handling JavaScript-heavy websites. Under this model, Googlebot receives prerendered pages, while regular users continue to access dynamic JavaScript content. Google has specifically endorsed this practice as a way to ensure that search engines can properly index content that would otherwise be difficult to crawl due to JavaScript rendering issues.

Prerender.io implements Dynamic Rendering by rendering content for search engine bots, like Googlebot, and delivering it directly to them while serving the live, interactive JavaScript page to users. This ensures that search engines can properly index your content and gives you a better chance of ranking higher in search results.

Here's a slide from Google's presentation:

Why It Matters

Prerender.io’s service aligns with Google's recommendation for dynamic rendering, and by using it, you can improve your site's SEO performance. Serving search engines with prerendered content ensures that:

• Search engines can crawl and index your content: Even if your site uses JavaScript for rendering, bots will still get access to fully-rendered pages.

• Better ranking potential: Pages that are properly indexed are more likely to rank well in search results.

• Avoid cloaking issues: Google permits sending prerendered pages to search engines by checking their user agents, avoiding the risk of cloaking (deliberately showing different content to users and search engines).

Prerender.io's middleware is designed to make the process seamless, ensuring that Googlebot and other major search bots receive the correct, fully-rendered version of your content.

 

Solution: Adding Googlebot to Your Middleware

If you're using Prerender.io, the integration is typically automatic, as Googlebot is included in the list of crawlers we check. If you need to manually add Googlebot (or other crawlers) to your middleware or if you’re using your own server setup, follow the instructions below:

1. Using nginx

In your nginx configuration, add the following line to check for Googlebot and other major crawlers:

if ($http_user_agent ~* "googlebot|bingbot|yandex|baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
    set $prerender 1;
}

 

2. Using Apache

In your Apache config, add the following condition to check for Googlebot, Bingbot, and Yandex:

RewriteCond %{HTTP_USER_AGENT} googlebot|bingbot|yandex|baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator [NC,OR]

 

3. Using Prerender-node

If you're using prerender-node, modify the user agent array like this:

var prerender = require('prerender-node').set('prerenderToken', 'YOUR_TOKEN');
prerender.crawlerUserAgents.push('googlebot');
prerender.crawlerUserAgents.push('bingbot');
prerender.crawlerUserAgents.push('yandex');
app.use(prerender);

 

4. Using Prerender-rails

If you're on Prerender for Rails, update the crawler_user_agents array:

ruby config.middleware.use Rack::Prerender, prerender_token: 'YOUR_TOKEN', crawler_user_agents: [
    'googlebot',
    'bingbot',
    'yandex',
    'baiduspider',
    'facebookexternalhit',
    'twitterbot',
    'rogerbot',
    'linkedinbot',
    'embedly',
    'bufferbot',
    'quora link preview',
    'showyoubot',
    'outbrain',
    'pinterest/0.',
    'developers.google.com/+/web/snippet',
    'www.google.com/webmasters/tools/richsnippets',
    'slackbot',
    'vkShare',
    'W3C_Validator',
    'redditbot',
    'Applebot',
    'WhatsApp',
    'flipboard',
    'tumblr',
    'bitlybot',
    'SkypeUriPreview',
    'nuzzel',
    'Discordbot',
    'Google Page Speed',
    'Qwantify'
]

 

5. Using ASP.NET MVC

For ASP.NET MVC, modify your web.config to include Googlebot, Bingbot, and Yandex:

<prerender token="YOUR_TOKEN" crawlerUserAgents="googlebot,bingbot,yandex"></prerender>

 

6. Using Prerender-java

In your web.xml, add Googlebot, Bingbot, and Yandex:

<filter>
    <filter-name>prerender</filter-name>
    <filter-class>com.github.greengerong.PreRenderSEOFilter</filter-class>
    <init-param>
        <param-name>prerenderToken</param-name>
        <param-value>YOUR_TOKEN</param-value>
    </init-param>
    <init-param>
        <param-name>crawlerUserAgents</param-name>
        <param-value>googlebot,bingbot,yandex</param-value>
    </init-param>
</filter>

 

Verify Your Installation

After integration, it’s important to verify that everything is working as expected. We’ve provided an easy-to-follow guide to help you test your setup and confirm that Prerender is functioning properly.

 

 

Related Articles / FAQs:

• What Prerender.io does
• How to add additional bots (integrations maintained by Prerender)