Make sure you are returning the correct status codes to crawlers.
We have two special meta tags to put in the <head> to let you return a different status code or header to a crawler depending on your REST calls since your original HTML page will always return a 200.
For example, if your REST endpoint returns a 404, set this meta tag and we'll return a 404 to the crawler:
<meta name="prerender-status-code" content="404">
For example, if your REST endpoint returns a 301, set this meta tag and we'll return a 301 and a Location header to the crawler:
<meta name="prerender-status-code" content="301"> <meta name="prerender-header" content="Location: http://www.example.com">
Tell us exactly when your pages are ready to be saved.
We try our very best to detect when a page is done loading by counting the number of requests in flight and saving the page when the number of requests in flight reaches zero (after a slight delay). You can tell us when your pages are ready so that we can be more accurate.
Put this in your HTML as early as you can (preferably the top of the <head> section):
<script> window.prerenderReady = false; </script>
When we see window.prerenderReady set to false, we'll wait until it's set to true to save the HTML. Execute the following when your page will be ready (usually after ajax calls).
window.prerenderReady = true;
Use our API to cache/recache pages.
You can use our API to cache pages when they are created to make sure they are cached when a crawler accesses them.
You can use our API to recache pages when they change on your end so that they are up-to-date without having to wait for the cache freshness value that you have set.
You can use our API to save on costs by setting your cache freshness to a high value and solely using the API to recache your pages.
Click here to view our /recache endpoint