Skip to content
Contact us
  • There are no suggestions because the search field is empty.

CDN Issues

Ensure bot requests bypass CDN cache so they reach your Prerender middleware for proper rendering.

 

Overview

While CDNs like Cloudflare, CloudFront, and others can significantly improve site speed and performance, they can unintentionally interfere with Prerender.io if not properly configured.

In particular, a CDN may serve cached (unrendered) content to bots like Googlebot—before the request ever reaches your origin server and Prerender middleware.

 

Here is what happened in detail:

Here’s a common scenario we've encountered:

  1. The customer has a website running behind a CDN.

  2. Prerender.io middleware is installed on the Apache or Nginx origin server.

  3. A search engine bot (e.g., Googlebot) visits the site.

  4. The CDN recognizes the requested page is cached.

  5. The CDN returns the cached content—without contacting the backend server.

  6. As a result, the bot receives raw JavaScript instead of the prerendered HTML.

This behavior prevents Prerender.io from doing its job, and search engines may index incomplete content—negatively affecting SEO.

 

What could be a solution?

The solution is to configure your CDN to be HTTP header-aware and bot-aware.

Specifically, configure your CDN to:

  • Detect known search engine bots via their User-Agent headers

  • Bypass cache for these bot requests

  • Allow the requests to pass through to the origin server, where the Prerender middleware can render the page correctly

This ensures bots receive the prerendered HTML instead of stale, cached JavaScript.

 

Configuration Tips (Per CDN)

Cloudflare

  • Use Cloudflare Workers or Page Rules

  • Match User-Agent headers

  • Set Cache Level: Bypass for bots

  • Optionally add a custom header like x-prerender: true for debugging

AWS CloudFront

  • Use a Cache Policy that varies by User-Agent

  • Or, implement Lambda@Edge to detect bots and modify behavior dynamically

Fastly

  • Write VCL logic to inspect the User-Agent header

  • Bypass cache or reroute to Prerender when a bot is detected

 

💡 Tips & Notes

  • Debug with Headers: Add a response header like x-prerender-status to verify whether a request was prerendered.

  • Keep the Bot List Updated: Bots evolve—check the Prerender bot list regularly.

  • Avoid Double Caching: Don’t cache prerendered content again in the CDN unless you control cache keys carefully.

  • Monitor SEO Crawl Reports: Use tools like Google Search Console to catch any indexing issues early.

  • Test Before Deploying: Try Prerender with and without CDN enabled to ensure bot traffic behaves as expected.

 

Get Support

Still have questions or need help? We’re here for you!

If you’ve followed the troubleshooting steps and still can’t resolve the issue, feel free to reach out to our support team. You can contact us via:

To help us resolve your issue as quickly as possible, please gather and include any relevant information, such as:

  • Error messages you're seeing
  • Steps you've already taken to troubleshoot
  • Screenshots or screen recordings (if applicable)

Providing these details up front will help our team diagnose the problem more efficiently and get you back on track faster.

 

Related Articles / FAQs: