Robots meta tag
Tell AI crawlers and search engines exactly how to treat each page — whether to index it, follow its links, or cache it.
TL;DR
The robots meta tag is an HTML directive placed in your page's <head> that tells AI crawlers and search engines how to treat that page. You can use it to prevent indexing, stop link signals from flowing, or block page caching. Without it, crawlers apply default behavior, which may index pages you didn't intend to make public.
Why the robots meta tag matters
AI crawlers and search engines apply default crawling and indexing behavior when a page has no robots tag. That default behavior can create problems you didn't intend:
- Unintended indexing. Pages like admin panels or thank-you pages may appear in search results.
- Uncontrolled link signals. Without
nofollow, pages you don't want to boost may inherit link authority. - Outdated cached versions. Without
noarchive, some crawlers may store cached versions of your pages. Note that Google no longer surfaces cached links, so this directive's effect varies by crawler.
Pages that contribute to your AI & Search visibility should remain indexable and followable. For pages with limited value to AI crawlers and search engines — login pages, cart flows, expired promotions — the robots meta tag lets you control what gets indexed and how.
The directives explained
| Directive | What it does |
|---|---|
index |
Allows the page to be indexed (default behavior if tag is absent) |
noindex |
Prevents the page from being indexed |
follow |
Allows crawlers to follow links on the page (default behavior) |
nofollow |
Prevents crawlers from following links on the page |
noarchive |
Historically prevented crawlers from caching this page. Google no longer supports cached links, so this directive has no effect on Google Search. Behavior on other crawlers varies. |
nosnippet |
Prevents crawlers from showing a content snippet in search results |
You can combine directives in a single tag. Directives are applied together, not in sequence.
Combining conflicting directives (for example, both index and noindex) is unnecessary and confusing. The more restrictive directive wins, so noindex will always override index. Use only one indexing directive and one link-following directive per tag.
How to add the robots meta tag
Place the tag inside the <head> section of your page's HTML:
<meta name="robots" content="index, follow"> To restrict a page, customize the content attribute:
<meta name="robots" content="noindex, nofollow, noarchive"> The tag applies only to the page it appears on. To block crawlers from an entire section of your site, use a robots.txt file instead.
Want to allow crawling but prevent indexing? Use noindex, follow. This lets AI crawlers and search engines discover links on the page without adding the page itself to their index.
Common use cases
| Page type | Recommended directive |
|---|---|
| Public content pages | index, follow (or omit the tag entirely) |
| Admin panels, internal tools | noindex, nofollow |
| Thank-you or confirmation pages | noindex, follow |
| Expired promotions or seasonal pages | noindex, nofollow |
| Pages with sensitive or duplicate content | noindex, nofollow, noarchive |
Validate your changes
After adding or updating your robots tag, use Google Search Console to confirm AI crawlers and search engines interpret the directives as intended. The URL Inspection tool shows you exactly how Googlebot reads a given page.
Prerender.io serves pre-rendered HTML to AI crawlers and search engines. The robots meta tag you set in your JavaScript app is preserved in the rendered output, so crawlers receive it correctly. See How Prerender.io works for the full request flow.
Related articles
- How Prerender.io works
- How the Cache Manager works
- Verify your Prerender.io integration
- Google's guide to the robots meta tag
Still need help? If you're unsure which directives to use for a specific page type or want to check that Prerender.io is passing your robots tag correctly to crawlers, our support team can help. → Contact us at support@prerender.io