Troubleshooting your Prerender.io integration and best practices

How Prerender.io routes crawler requests

Understanding the request flow helps you pinpoint where things break.

1. An AI crawler or search engine bot (such as GPTBot or Googlebot) requests your page.
2. Your integration detects the bot's user agent and forwards the request to Prerender.io.
3. Prerender.io fetches your page, renders the JavaScript in a headless browser, and caches the resulting HTML. Static assets (images, CSS, and fonts) are not cached.
4. Prerender.io returns the rendered HTML to your integration.
5. Your integration serves the rendered HTML to the bot. The bot then fetches static assets directly from your server.

If any step in this chain is broken, the bot receives incomplete or no content.

Common issues and fixes

Bots blocked by your server

What happens: Your server rejects requests from bot user agents before Prerender.io can intercept them.

Fix: Check your firewall rules and .htaccess configuration. Ensure the user agents listed in What user-agent strings and IP addresses does Prerender.io use? are allowed through. Enable any required server modules, for example headers_module and proxy_module for Apache .htaccess rules. Also check for conflicting rewrite rules in .htaccess or your server configuration.

Middleware in the wrong order

What happens: The Prerender.io middleware is attached after other middleware that handles JavaScript or API responses, so requests never reach it.

Fix: Move the Prerender.io middleware to the front of your middleware stack, before any middleware that serves JavaScript or API routes. Check your configuration against the integration guides for your framework.

Geo or IP-based blocking

What happens: Prerender.io operates servers worldwide. If your site uses geo-based filtering or IP allowlists, a Prerender.io server in a different region may be blocked from fetching your page.

Fix: Whitelist Prerender.io's IP ranges and user agents in your firewall settings. See How can I whitelist Prerender.io's IP addresses and user agents? for the current list.

Staging sites blocking Prerender.io

What happens: Password-protected or IP-restricted staging sites block Prerender.io from accessing your pages. Prerender.io runs on the public internet and must be able to reach your site to render it.

Fix: Add Prerender.io's user agents and reverse-lookup IP addresses to your staging site's allowlist. A smart firewall that supports reverse DNS lookup makes this straightforward.

CDN blocking Prerender.io's user agent

What happens: Some CDNs block requests from non-standard user agents, including Prerender.io's rendering bot, which prevents pre-rendering from working.

Fix: Add Prerender.io's user agents to your CDN's allowlist. After making the change, simulate a request using the Prerender.io user agent and check your browser console for JavaScript errors to confirm the block is resolved.

Unwanted URLs being cached

What happens: Prerender.io caches URLs with query parameters or unnecessary variations that do not affect page content, leading to bloated cache usage.

Fix:

• Use the URL Parameters feature in your Prerender.io dashboard to exclude query parameters that do not change the rendered output.
• Configure your integration to forward only the pages or sections of your site that need pre-rendering.
• Ensure that invalid or unnecessary pages return a non-200 status code from your server (for example, 404 for missing pages). Alternatively, use a Prerender.io meta tag to instruct Prerender.io to return a non-200 status code after JavaScript execution for specific URLs.

Prerender.io returns raw or empty content

What happens: Prerender.io returns a response containing the X-Prerender-Raw-Data header. This means Prerender.io could not render the page and is returning the original unrendered content instead.

Why this happens: Prerender.io always attempts to serve a page version. When rendering fails, it falls back to the raw source rather than returning an error, so the bot still receives a response.

Fix: See What should I do if my pages are not being cached properly? for a step-by-step diagnosis walkthrough.

How to test your prerendered pages

Use any of these methods to confirm your pages are rendering correctly.

Option 1: Check cached versions in your dashboard See How to check cached versions of your URLs? and confirm the response contains fully rendered HTML without unnecessary JavaScript tags.

Option 2: curl or Postman Fetch your page using curl or Postman, setting the user agent to match Prerender.io's rendering bot.

Option 3: Override user agent in Chrome Use Chrome DevTools (Network conditions panel) to override the user agent, or install a user-agent switcher extension. Set the user agent to Googlebot and request your page.

Related articles