How we respond to bots
We return the same status code that you return to us by default. There are some exceptions when we change your status code. You can read about them in the Status Code Mapping section. These are the status codes that you can see on the Crawler visits page, in your Dashboard.
When everything is working OK, or your page was cached before and the current content is being served.
400 (Bad Request)
When our service receives an invalid URL.
A wrong token has been provided.
This code is shown if the token is not provided.
429 (Too Many Requests)
You will see this code if you are on the basic plan and hit the plan's limit.
503 (Service Unavailable)
If the user is canceled, this code will be shown when trying to use their token.
504 (Gateway Timeout)
We return 504 in the following cases:
- when your page is blank for some reason e.g:
- If an error happens while rendering your page
- any other error happens on our side
Bots will try again if they get a 504.
Status Code mapping
These are the status codes we change to improve your SEO:
304 (Not Modified) to 200 (OK)
Whenever you return a 304 (Not Modified) to us, we will respond to the bot with a 200 (OK).
Timeout status code
By default, we respond with a 504 (Gateway Timeout) if the render takes longer than the timeout (20 seconds by default). You can change the status code that should be returned to the bot if a render takes more than the default timeout.
prerender-status-code meta tag
You can set up the status code that should be returned to a bot with a meta tag. In the Best Practices section, you can read more about it.