Adding Googlebot to your integration

Google recommends that you use Prerender.io in their Dynamic Rendering documentation.

In May of 2018, Google introduced "Dynamic Rendering" for serving normal JavaScript to your users and serving Prerendered pages to search engines. This is exactly what we do at Prerender.io.

The change introduced by Google for Dynamic Rendering means they will stop crawling the ?_escaped_fragment_= URLs and you can now serve a prerendered page to Googlebot by checking their user agent directly. That might sound like cloaking, but Google introduced a policy change where they are allowing you to send prerendered pages to Googlebot by checking their user agent.

Here's a slide from Google's presentation:

Due to this Dynamic Rendering announcement, we have changed our middleware to add Googlebot to the list of user agents being checked directly.

If you are using our Prerender.io hosted service you won't need to make any code changes besides upgrading the Prerender.io middleware to make sure it's checking Googlebot by their user agent.

Here is the relevant part of the video of Google's Dynamic Rendering presentation:

Manually Adding Googlebot to your middleware

If you don't want to upgrade yet, you can easily add Googlebot to your middleware

nginx

Find the line where you are checking for user agents and add googlebot, bingbot, and yandex

if ($http_user_agent ~* "googlebot|bingbot|yandex|baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") { 	set $prerender 1; }

Apache

Find the line where you are checking for user agents and add googlebot, bingbot, and yandex

RewriteCond %{HTTP_USER_AGENT} googlebot|bingbot|yandex|baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator [NC,OR]

prerender-node

Where you are using prerender-node, just add the user agents directly to the array of user agents being checked

var prerender = require('prerender-node').set('prerenderToken', 'YOUR_TOKEN');
prerender.crawlerUserAgents.push('googlebot');
prerender.crawlerUserAgents.push('bingbot');
prerender.crawlerUserAgents.push('yandex');
app.use(prerender);

prerender_rails

Pass in a new array of user agents with googlebot, bingbot, and yandex in the array

ruby config.middleware.use Rack::Prerender, prerender_token: 'YOUR_TOKEN', crawler_user_agents: [
	'googlebot',
	'bingbot',
	'yandex',
	'baiduspider',
	'facebookexternalhit',
	'twitterbot',
	'rogerbot',
	'linkedinbot',
	'embedly',
	'bufferbot',
	'quora link preview',
	'showyoubot',
	'outbrain',
	'pinterest/0.',
	'developers.google.com/+/web/snippet',
	'www.google.com/webmasters/tools/richsnippets',
	'slackbot',
	'vkShare',
	'W3C_Validator',
	'redditbot',
	'Applebot',
	'WhatsApp',
	'flipboard',
	'tumblr',
	'bitlybot',
	'SkypeUriPreview',
	'nuzzel',
	'Discordbot',
	'Google Page Speed',
	'Qwantify'
]

ASP.NET MVC

In your web.config, add googlebot, bingbot, and yandex to the crawlerUserAgents in the prerender configuration

<prerender token="YOUR_TOKEN" crawlerUserAgents="googlebot,bingbot,yandex"></prerender>

prerender-java

In your web.xml, add googlebot, bingbot, and yandex to the crawlerUserAgents in the prerender configuration

<filter>
	<filter-name>prerender</filter-name>
	<filter-class>com.github.greengerong.PreRenderSEOFilter</filter-class>
	<init-param>
		<param-name>prerenderToken</param-name>
		<param-value>YOUR\_TOKEN</param-value>
	</init-param>
	<init-param>
		<param-name>crawlerUserAgents</param-name>
		<param-value>googlebot,bingbot,yandex</param-value>
	</init-param>
</filter>