Why is my CDN serving unrendered content to bots?

TL;DR

When a CDN caches your pages and serves them directly to incoming requests, AI crawlers and search engines never reach your Prerender.io integration on the origin server. They receive raw JavaScript instead of rendered HTML. Configuring your CDN to bypass the cache for known bot User-Agent headers resolves the problem.

Why CDNs can block Prerender.io

CDNs like Cloudflare, CloudFront, and Fastly are designed to serve cached content as quickly as possible. That's useful for your website visitors, but it creates a problem for AI crawlers and search engines when your Prerender.io integration sits on the origin server.

Here's what happens:

A bot such as Googlebot or GPTBot visits your site.
The CDN finds the requested page in its cache.
The CDN returns the cached content without contacting your origin server.
The bot receives raw JavaScript, not rendered HTML.
Prerender.io never receives the request and cannot serve the rendered version.

The result: AI crawlers and search engines may index incomplete or empty content from your site.

⚠️ This issue can occur even when your Prerender.io integration is correctly configured. If your CDN sits in front of your origin server and is not bot-aware, rendered pages will not reach AI crawlers.

How to fix it: configure cache bypass for bots

The fix is to configure your CDN to detect bot User-Agent headers and bypass the cache for those requests, passing them through to your origin server where Prerender.io can render and respond correctly.

ℹ️ Not sure how to configure your specific CDN? Ask Nexus, your AI integration assistant, inside your Prerender.io dashboard. Describe your setup and Nexus will walk you through the right configuration.

Cloudflare

Use Cloudflare Workers or Page Rules to inspect the User-Agent header and set Cache Level: Bypass for matching bot requests. You can optionally add a custom response header such as x-prerender: true to confirm the bypass is active and help with debugging.

AWS CloudFront

Configure a Cache Policy that varies responses by User-Agent. Alternatively, use Lambda@Edge to detect bots and modify caching behavior dynamically. See the CloudFront integration guide for full configuration steps.

Fastly

Write VCL logic to inspect the User-Agent header. When a known bot is detected, bypass the cache or reroute the request to your origin server. See the Fastly integration guide for full configuration steps.

Verifying your fix

Once you've applied CDN bypass rules, verify that bot requests are reaching Prerender.io correctly.

Add a response header. Include a header such as x-prerender-status in your responses to confirm whether a page was pre-rendered by Prerender.io.
Check your Prerender.io dashboard. Cached pages typically appear within 2 to 5 minutes of a bot request reaching your integration, though this can vary.
Monitor your crawl reports. Use Google Search Console to check for indexing errors that may indicate bots are still receiving unrendered content.
Keep bot User-Agents current. AI crawlers are updated regularly. Review the Prerender.io crawler list periodically to ensure your bypass rules cover current bot User-Agents.

ℹ️ Avoid caching pre-rendered content a second time at the CDN layer unless you control cache keys carefully. Double caching can cause AI crawlers to receive stale rendered HTML.

➡️ See How to verify your Prerender.io integration for step-by-step verification instructions.

💬 Still need help? If you're still seeing CDN caching issues after applying these steps, our support team can help you diagnose the configuration. → Contact us at support@prerender.io