How to Avoid 429 Error Codes with Prerender

TL;DR

A 429 (Too Many Requests) response means crawlers are being told there are too many requests to fulfill right now. On Prerender, 429s usually come from one of three causes: you've exhausted your 30-day trial allowance (25,000 renders plus 15% headroom), your own server or firewall is rate-limiting Prerender's crawler, or aggressive recaching is overwhelming your origin. Each cause has a different fix — the sections below cover all three.

Why this matters

AI crawlers and search engines like GPTBot, ClaudeBot, and Googlebot rely on receiving fully rendered HTML to index your content. When they get a 429 instead, indexing slows or stops — which can hurt your visibility in search results and AI citations.

If 429s persist, crawlers may reduce their crawl frequency over time, compounding the problem. Resolving them quickly keeps the flow of clean HTML to crawlers open, so your pages stay readable, indexable, and citable.

How to check your current usage

Open the Billing section in your Prerender dashboard to see your current render usage, plan limit, and remaining headroom. This is the fastest way to confirm whether the 429s are caused by your render allowance or something else.

Cause 1: You've exhausted your trial render allowance

Prerender's Starter plan includes a 30-day free trial with 25,000 renders per month plus a 15% headroom buffer. Once you've used that combined allowance during your trial, Prerender returns 429 responses to crawler requests for uncached pages until your trial converts to a paid plan or the next billing cycle starts.

How to fix it: Upgrade to a paid plan that fits your traffic volume. Current paid plans are:

• Starter — 25,000 renders
• Growth — 100,000 renders
• Pro — 500,000 renders
• Enterprise Plus — 1M+ renders (custom pricing)

Paid plans continue to render uncached pages after you reach your included limit and apply overage fees per 1,000 renders, so crawler access isn't interrupted. See current pricing and overage rates for details.

Cause 2: Your server, firewall, or CDN is rate-limiting Prerender's crawler

If you're already on a paid plan and still seeing 429s, the limits are coming from your own infrastructure rather than from Prerender. Common culprits are firewall rules, WAF configurations, DDoS protection thresholds, and proxy-level rate limits that throttle Prerender's crawler when it requests many pages in a short window.

How to fix it:

• Check your firewall, WAF, and reverse proxy logs for blocked or throttled requests from Prerender's crawler IPs.
• Confirm that DDoS protection isn't treating Prerender's batch rendering as a threat.
• Whitelist Prerender's IP addresses and crawler user-agent. See Prerender IP addresses or pull the full list from ipranges.prerender.io/ipranges-all.txt.

Cause 3: Aggressive recaching is overwhelming your origin

If your cache expiration is set too short, Prerender re-renders pages frequently. That can flood your origin server with requests, which then triggers your own rate-limiting protections — producing 429s on the cache-refresh requests.

How to fix it:

• Increase your cache expiration in the dashboard. Set it based on how often your content actually changes — a product catalog updated daily doesn't need a 6-hour cache.
• Selectively refresh only updated pages using the /recache API instead of waiting for full cache expiration cycles.
• Use ignore URL filters to skip low-priority pages that don't need rendering.

Additional tips and best practices

• 429s are SEO-critical. If AI crawlers and search engines receive them frequently, they may slow down or halt crawling, compounding the visibility loss.
• Match cache duration to content velocity. Pages that change weekly don't need hourly recaching. Tune cache duration to your actual update cadence.
• Filter out website visitor traffic at the integration layer. Your integration should only route crawler requests to Prerender — serving normal users through Prerender wastes your render allowance.
• Use regex filters and ignore URL lists. Skip rendering low-priority pages (admin areas, search results, faceted URLs) that don't need to appear in search.

How to verify your fix worked

After applying a fix, monitor your dashboard logs and your server's response codes for crawler requests. If 429s have stopped and you're seeing successful responses (200 with Prerender headers), the fix is working.

For a full verification walkthrough, see How to test your site after integration.

Related articles

• • Prerender status codes explained
  • What should I do if I reach my render limit?
  • Has your rendering paused?
  • Prerender IP addresses to whitelist