Skip to content
Contact us
  • There are no suggestions because the search field is empty.

Robots meta tag

Control how search engines crawl, index, and cache your web pages to improve SEO and protect sensitive content.

The robots meta tag is used to give search engine crawlers specific instructions on how to treat your web page. It allows you to manage whether a page should be indexed, whether links on the page should be followed, and whether a cached version should be stored. Proper use of this tag is critical for maintaining search visibility, avoiding duplicate content issues, and keeping non-public pages out of search results.

Detailed Explanation / How It Works

Why the Robots Meta Tag Matters

When a page lacks a robots tag, search engines will apply default crawling and indexing behavior. This can lead to unintended consequences such as:

  • Unintended Indexing: Pages like admin panels or thank-you pages may show up in search results.
  • Uncontrolled Link Flow: Without nofollow, pages you don’t want to boost might inherit link signals.
  • Cached Content: Without noarchive, outdated versions of your pages might be stored and shown in search results.

Recommended Use Cases

Pages that contribute to your SEO rankings should generally remain indexable and followable. But for pages with limited value to search engines (e.g. login pages, cart flows, expired promotions), the robots meta tag allows you to control visibility and avoid polluting search results.

Step-by-Step Usage

1. Add the Robots Tag to Your HTML

Place the tag in the <head> section of your page:

<meta name="robots" content="index, follow">

2. Customize Directives as Needed

Change the content attribute to meet your goals. Common instructions include:

  • noindex – Prevent the page from being indexed
  • nofollow – Prevent crawlers from following links on the page
  • noarchive – Prevent caching of this page by search engines
  • nosnippet – Prevent search engines from showing content snippets in results

Example with multiple directives:

<meta name="robots" content="noindex, nofollow, noarchive">

3. Validate with Tools

After adding or updating your robots tag, use tools like Google Search Console to confirm bots interpret directives as intended.

Common Pitfalls / Tips

  • Forgetting the robots tag on staging or thank-you pages can lead to unintended indexing.
  • If you want bots to crawl but not index, use noindex, follow.
  • Combining conflicting directives (e.g., index and noindex) can lead to unpredictable behavior.

Related Articles