CDN related issues
CDNs (e.g. Cloudflare, Cloudfront...etc) caching your site content can cause hard to identify issues. We've seen cases when the customer had a CDN hosted in front of their website, and it intercepted the request before it could even make its way to the Prerender middleware.
Here is what happened in details:
- The customer had a well working website with CDN added.
- Customer installed Prerender on their apache/nginx server, expecting it to prerender pages for bots.
- Random bot (e.g. Google bot) visited the page
- The page that was requested existed in the CDN cache, so the CDN responded not even invoking the backend servers (including the Prerender middleware)
- The bot received an unrendered response.
What could be a solution?
The CDN must be HTTP header aware and not responding for those user agents that you want your site to be prerendered for.