Fastly integration
  • 20 Jul 2022
  • 1 Minute to read
  • Contributors
  • Dark
  • PDF

Fastly integration

  • Dark
  • PDF

Article Summary

Fastly integration's principle is to add as a new origin and serve the content from it when a bot request arrives.


You need a condition to tell Fastly when to get the content from

Go to Service Configuration -> Conditions and create a new condition:


The Apply if... rule should be like this:

req.http.User-Agent ~ "(?i)googlebot|bingbot|yandex|baiduspider|facebookexternalhit|twitterbot|linkedinbot|embedly|showyoubot|outbrain|pinterestbot|slackbot|vkShare|W3C_Validator|whatsapp|ImgProxy| flipboard|tumblr|bitlybot|skype|nuzzel|discordbot|google|qwantify|pinterest|lighthouse|telegrambot" && req.url.ext !~ "(?i)js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|woff|ttf|svg|webmanifest" 

Please note that some rare crawlers have been removed from these rules due to the conditions' size limitations.

Origin host

Go to Service Configuration -> Origins -> Hosts.

Add as a new origin host:

You also need to attach this to the condition you created above:


This will ensure we serve the content from if a crawler visits your site.

Setting up header rules

Header rules' function is to send your API token to and rewrite the URL to its appropriate form. Due to this, you are going to need two rules.
Go to Service Configuration -> Content -> Headers

API token

Create a new header.

The Destination should be http.x-prerender-token and the Source is your API token, which you can find in your Dashboard

Please note you need to wrap the token in quotation marks (").

Attach the rule to the condition:


URL rewrite

Create a new header. Note that instead of modifying headers, this rule will modify the URL.
The Destination should be url and the Source is "/https://" req.url.

Attach the rule to the condition:


Optional: User-agent forwarding

This rule will copy the user-agent header from the incoming request to the request sent to prerender. This way you'll see the actual user-agent on the crawl stats page.

Create a new header. The Destination is going to be http.user-agent and the Source is req.http.user-agent
You need to attach the rule to the same condition

Once you are done, you header rules should look something like this:


After activating your configuration, it should work.
We usually recommend testing your setup using curl
With this script, you can send a request mimicking the Google crawler.

curl  -v -A googlebot

If the response contains the x-prerender-requestid header, the request went through
You should be able to see the requests on the Dashboard on the Crawler Visits page

And that's it!

Was this article helpful?

What's Next