Akamai

This article covers the basics of a prerender service and explains how to get started.

You can easily add the Prerender.io service to your delivery by adding a new rule:

1. Detect crawlers
2. Send crawler requests to prerender service
3. Modify request path
   
   

Detect crawlers

Detecting the key crawlers can be done based on the user-agent request header. (If you use Bot Manager, you’ll have more flexibility and better results).

Add a Criteria that checks if the User Agent is one of the bots' below.
You can find a list of these search bots below.

 'Google-InspectionTool',
  'googlebot',
  'Yahoo! Slurp',
  'bingbot',
  'yandex',
  'baiduspider',
  'facebookexternalhit',
  'twitterbot',
  'rogerbot',
  'linkedinbot',
  'embedly',
  'quora link preview',
  'showyoubot',
  'outbrain',
  'pinterest/0.',
  'developers.google.com/+/web/snippet',
  'slackbot',
  'vkShare',
  'W3C_Validator',
  'redditbot',
  'Applebot',
  'WhatsApp',
  'flipboard',
  'tumblr',
  'bitlybot',
  'SkypeUriPreview',
  'nuzzel',
  'Discordbot',
  'Google Page Speed',
  'Qwantify',
  'pinterestbot',
  'Bitrix link preview',
  'XING-contenttabreceiver',
  'Chrome-Lighthouse',
  'TelegramBot',
  'SeznamBot',
  'Perplexity',
  'OAI-SearchBot',
  'ChatGPT',
  'GPTBot',
  'ClaudeBot',
  'Amazonbot',

Send crawler requests to prerender service

Whenever the user agents are detected, we forward them to `service.prerender.io`. 

Add a Behavior for Origin Server, make sure to use these key settings:

Origin Type: Your origin

Origin Server Hostname: service.prerender.io

Forward host header: Origin Hostname

Cache key hostname: Origin Hostname

Add your Prerender token using the Modify Outgoing Request header behavior.

In practice, this means that you need to set the x-prerender-token header in the forwarded request to the token. Your Prerender.io Token can be found in the Security and Access menu.

Modify request path

We still need to change the outgoing request path to the full URL. You can do this by using property variables and modifying the outgoing request path behavior.

Set the Action to Replace the entire path and add the following to the Replace with field.

/://

Do not cache

Crawlers typically generate a lot of traffic. Prerender visits each page only once per crawl; by the time our service crawls the same page for the second time, the cached version is either evicted or no longer up to date.

So, checking the edge cache, the parent cache, and going forward to the origin is slower than going straight to the origin. Going to the origin gives you the benefits of SureRoute and persistent connections.