Overview
Prerender's Sitemap Crawler helps search engines index your site faster by discovering and caching sitemap URLs. Adding sitemaps to your Prerender account ensures better SEO and performance, especially for large or new sites.
How It Works
- Automatic Discovery: Prerender sitemap crawler automatically finds your sitemaps. If it doesn’t, you can manually add them to ensure all URLs are cached and ready for search engines.
- Efficient Caching: URLs from the sitemap are added to the manual/API rendering queue and cached, so bots receive faster responses when they visit your site.
- Regular Checks: Sitemaps are checked and updated every 7 days, daily, or hourly depending on the interval you want. New URLs are added to the queue and cached automatically.
- Multiple Sitemaps Supported: Perfect for managing complex site structures.
- Standard Format: Supports XML sitemaps as per sitemaps.org.
Process
- Prerender.io downloads the site map from the server using the Prerender user agent.
- Adds < url >...< /url > URLs to the cache (if the URL is already in the cache, it skips this step).
- Downloads and processes the additional < sitemap>...</ sitemap> sitemaps.
How to Use It
Add a Sitemap
- Go to the "Sitemaps" menu in your dashboard.
- Click Import Sitemap at the top right.
- Enter your sitemap URL.
- Choose cache options (desktop/mobile) and re-crawl intervals.
- Click Import to start caching.
Important note: When you click the Import button while adding the sitemap, the caching process will begin automatically for all associated URLs.
Adjust Crawling Settings
- Click the 👁 icon next to a sitemap you want to adjust.
- Edit revisiting intervals and rendering devices preferences.
- Recrawl manually from the dashboard or via the /sitemap API endpoint.
Recrawling Process
- When adding a sitemap, caching begins automatically for all listed URLs.
- Recrawl settings apply to subsequent crawls and can be customized per sitemap.
- Use the /sitemap API endpoint to integrate automatic recrawls into your workflow whenever the sitemap is updated.
Tips & Troubleshooting
Possible reasons why the sitemap might be flagged as Unhealthy:
- Check Your URL: Ensure it’s correct and accessible.
- Whitelist Prerender's IP addresses and user agents. Allow Prerender's crawler access.
- Fix Sitemap Issues: Follow sitemaps.org guidelines.
- Monitor Health: Look for the ✅ icon (healthy) or ❓ icon (pending).
- Address Errors: Common issues include incorrect URLs, blocked crawler, or non-200 HTTP status codes for sitemap entries.
Additional information
You can always check if your sitemaps are healthy and available for the Prerender sitemap crawler. If the Sitemap is available and can be crawled by our crawler, you will see a ✅ icon under the Health column. When initially adding it to the Sitemap menu, you will see a ❔icon under the health column. It might take a couple of minutes for the icon to change.
The Sitemap table will show you if the sitemap is not available or cannot be crawled.
You can also check when a sitemap was crawled the last time (Last visited at column) and when it will be crawled again (Next visit column) by clicking on the 👁️ icon.
The URLs that were already detected previously on the Sitemap will be skipped and won't be recached when the sitemap is re-crawled.