Skip to content
Contact us
  • There are no suggestions because the search field is empty.

General Tips and Best Practices

Prerender.io enables search engine bots to index fully rendered pages from JavaScript-heavy websites, boosting SEO and performance. However, issues can arise due to server misconfigurations, middleware order, or network restrictions. This guide explains common challenges and their solutions to help you optimize your integration.

Detailed Explanation / How It Works

To pinpoint issues, it’s essential to understand Prerender.io’s workflow:

  1. A bot (e.g., Googlebot) requests your page.
  2. The integration implemented on your end detects the bots's requests and forwards the request to Prerender.
  3. Prerender then fetches your page but it only renders, caches and stores the DOM, it doesn't cache images or any other static assets.
  4. Then, Prerender renders all JS content on your page and returns the fetched and prerendered DOM content to the middleware.
  5. Finally, Prerender sends the prerendered page back to the bot, which indexes the content and fetches static assets from your server.

What Can Go Wrong?

Here are common issues that can disrupt this process and their solutions:

1. Bots Blocked by Your Server

Issue: Your server may reject bot requests (e.g., Googlebot), preventing Prerender from intercepting them.

Solution:

  • Ensure bots like Googlebot aren’t blocked by your firewall or .htaccess rules.
  • Allow user agents listed in Prerender.io’s documentation.

2. Middleware Misconfiguration

Issue: Middleware attached in the wrong order can prevent Prerender from working properly. Prerender has to be set up into the part of your system that handles incoming requests.

Solution:

  • Attach the Prerender.io middleware before others that serve JavaScript or APIs.
  • Verify your configuration matches Prerender.io’s integration guidelines.

3. Geo or IP-Based Blocking

Issue: Geo-based filters or IP restrictions may prevent Prerender.io servers from accessing your site. Prerender.io servers are all around the globe. It may happen that a Prerender server hosted in Germany tries to fetch your geo-filtered US-based server, and it fails. As we only render and cache the HTML content, bots (like Googlebot) will still reach out to your server to fetch the rest of your page. This can still fail due to user agent or IP filtering.

Solution:

  • Allow Prerender.io’s IP ranges and user agents in your firewall settings.
  • Test access by simulating requests using Prerender.io’s user agent.

4. Staging Sites Blocked

Issue: Private staging sites may restrict Prerender.io’s access.

Solution:

  • Enable access for Prerender.io user agents. Prerender service is hosted on the public internet, you definitely have to allow it to access your site. 
  • Use a smart firewall to whitelist Prerender.io’s reverse-lookup IP addresses.

5. User-Agent Blocked by CDNs

Issue: Certain CDNs may block requests from non-standard user agents, including Prerender.io, which can prevent proper pre-rendering.

Solution:

  • Simulate a request using the Prerender.io user agent and inspect the browser console for JavaScript errors.
  • Allow Prerender.io's user agents in your CDN settings to ensure uninterrupted functionality.

Tips

  • Enable required server modules (e.g., headers_module, proxy_module) for .htaccess functionality.
  • Check for conflicting rewrite rules in .htaccess or your server configuration.

Common Pitfalls

Unwanted Links Cached

Issue: Excessive or irrelevant URLs, such as those with unnecessary query parameters, are being cached by Prerender.

Solution:

  • Use the URL Parameters feature to exclude query parameters that do not impact page rendering.

  • Modify the integration to send only certain page or section of your site to Prerender.

  • Ensure invalid or unnecessary pages return a non-200 status code from your server (e.g., 404 for missing pages). Alternatively, use a Prerender meta tag to instruct Prerender to return a non-200 status code after JavaScript execution for specific URLs. See this page for reference.

Prerender Returns Raw or Empty Content

If the prerender returns an X-Prerender-Raw-Data header, the crawler cannot render the page and returns the original content instead. This happens because we still want to serve a page version, even if we cannot render it.

How to Test Your Prerendered Pages:

Use the following methods to verify if your pages are prerendered correctly:

  1. Follow this guide and check that the page contains fully-rendered content without unnecessary JavaScript tags.
  2. Use curl or Postman to fetch your site with the Prerender.io user-agent.
  3. You can test it with a single browser using Chrome's built in feature to override the user agent. More info here. If that seems complicated, you can also install a user agent switcher (we suggest using this one), then set up a new user agent that looks like a bot (e.g. Googlebot), and fetch your page.

Related Resources: