- Print
- DarkLight
- PDF
Fastly integration's principle is to add service.prerender.io as a new origin and serve the content from it when a bot request arrives.
Condition
You need a condition to tell Fastly when to get the content from Prerender.io.
Go to Service Configuration
-> Conditions
and create a new condition:
The Apply if...
rule should be like this:
req.http.User-Agent ~ "(?i)googlebot|bingbot|yandex|baiduspider|facebookexternalhit|twitterbot|linkedinbot|embedly|showyoubot|outbrain|pinterestbot|slackbot|vkShare|W3C_Validator|whatsapp|ImgProxy| flipboard|tumblr|bitlybot|skype|nuzzel|discordbot|google|qwantify|pinterest|lighthouse|telegrambot" && req.url.ext !~ "(?i)js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|woff|ttf|svg|webmanifest"
Please note that some rare crawlers have been removed from these rules due to the conditions' size limitations.
Origin host
Go to Service Configuration
-> Origins
-> Hosts
.
Add service.prerender.io
as a new origin host:
You also need to attach this to the condition you created above:
This will ensure we serve the content from Prerender.io if a crawler visits your site.
Setting up header rules
Header rules' function is to send your API token to Prerender.io and rewrite the URL to its appropriate form. Due to this, you are going to need two rules.
Go to Service Configuration
-> Content
-> Headers
API token
Create a new header
.
The Destination
should be http.x-prerender-token
and the Source
is your API token, which you can find in your Prerender.io Dashboard
Please note you need to wrap the token in quotation marks ("
).
Attach the rule to the condition:
URL rewrite
Create a new header
. Note that instead of modifying headers, this rule will modify the URL.
The Destination
should be url
and the Source
is "/https://" req.http.host req.url
.
Attach the rule to the condition:
Optional: User-agent forwarding
This rule will copy the user-agent
header from the incoming request to the request sent to prerender. This way you'll see the actual user-agent
on the crawl stats page.
Create a new header
. The Destination
is going to be http.user-agent
and the Source
is req.http.user-agent
You need to attach the rule to the same condition
Once you are done, you header rules should look something like this:
Testing
After activating your configuration, it should work.
We usually recommend testing your setup using curl
With this script, you can send a request mimicking the Google crawler.
curl -v -A googlebot www.yourpage.com/
If the response contains the x-prerender-requestid
header, the request went through Prerender.io.
You should be able to see the requests on the Dashboard on the Crawler Visits page
And that's it!