How and When to Use a Noindex Tag: The Straight-Talking Guide


Run a Crawl Now

What a Noindex Tag Does—and What It Does Not

noindex is an indexing directive. When Googlebot can crawl a URL and read the directive, Google removes that page or file from Google Search results. For an HTML page, you normally place a robots meta tag in the page’s <head>. For a PDF, image, video, or another non-HTML resource, you normally send the directive in an X-Robots-Tag HTTP response header. Google documents both methods in its noindex implementation guidance.

Noindex does not make a URL private. The page can still be opened by anyone who has the link, discovered through other websites, recorded in analytics, or accessed by crawlers that do not support the directive. Confidential, paid, customer-specific, or otherwise sensitive content needs authentication or another genuine access control. Google’s removal guidance explicitly separates password protection and content removal from noindex.

The practical question is therefore not simply “Do I want this page to rank?” It is: “Should this URL remain publicly accessible, and if so, should it be eligible to appear in search?” Noindex is appropriate only when the answer is yes, keep it accessible and no, keep it out of search.

Before removing an indexing restriction for AI search, confirm that the page is intended to be public. A website audit can check the instruction, not whether a provider has indexed or mentioned the page. The permissions guide explains why search and training choices should remain separate.

The Noindex Decision in 60 Seconds

  • The page must stay publicly accessible, but it should not appear in search: use noindex.
  • The page is a duplicate or near-duplicate and you want signals consolidated into a preferred version: use a canonical rather than noindex.
  • The content has moved or has a clear replacement: use a permanent redirect to the relevant destination.
  • The content has gone and there is no suitable replacement: return a genuine 404 or 410 response.
  • The content is confidential, paid, personal, or limited to authorised users: remove it from public access or protect it with authentication. Noindex is not security.
  • Your aim is to reduce crawling of large numbers of low-value URL variations: treat that as a crawl-management and URL-inventory problem. Noindex still requires Google to request the URL before it can read the directive.

Do not noindex a page merely because it is short, old, or currently performs poorly. First decide whether the page should exist, whether it satisfies a useful purpose, and whether another URL should replace or represent it.

When Noindex Is the Right Choice

A good noindex candidate usually meets all four conditions below:

  1. The URL needs to remain available to users or business systems.
  2. It is not the preferred version of content that should rank.
  3. There is no better destination that users should be redirected to.
  4. Allowing it into search would create a poor, misleading, or unnecessary search result.

Common examples

  • Generic form-confirmation or thank-you pages: the page must load after a completed action, but it is not a useful landing page for searchers. Customer-specific confirmations should also be protected from unauthorised access.
  • Account utility pages with no public search value: for example, a generic sign-out confirmation or preference-completion page. Do not use noindex as a substitute for protecting the account itself.
  • Public files intended for direct distribution: a PDF may be safe for anyone to open but unnecessary in search results. An X-Robots-Tag: noindex response can address the indexing decision without pretending the file is private.
  • Campaign or operational pages that must remain live: a URL may still support emails, adverts, QR codes, or customer journeys after it stops being an appropriate organic-search landing page.
  • Selected filter, search, or generated-result pages: noindex may be appropriate where the page must remain usable but should not become a search result. For large sites with extensive URL combinations, assess canonicalisation and crawl controls as well; noindex alone does not stop crawling.

Apply the decision page by page or to a tightly defined template. A broad rule such as “noindex every thin page” is risky because useful contact pages, calculators, product variants, legal pages, and concise answers can all be short for legitimate reasons.

When Noindex Is the Wrong Tool

Private or sensitive information

If a person should not be able to open the content without permission, use authentication, authorisation, removal, or another server-side access control. A noindexed page remains publicly reachable. This is especially important for member downloads, invoices, customer records, staging sites, unpublished documents, and files containing personal information.

Duplicate pages that should consolidate into one version

Use a canonical when multiple accessible URLs contain the same or very similar content and you want Google to treat one as the representative version. Google recommends canonical annotations rather than using noindex to prevent selection of an unwanted canonical. Our guide to diagnosing canonical issues explains the broader decision. Do not routinely place both noindex and a canonical on the same duplicate: one says “exclude this URL,” while the other asks Google to consolidate it with another URL.

Moved or deleted content

If a page has moved, redirect users and search engines to the relevant replacement. If it has permanently disappeared and there is no equivalent destination, return 404 or 410. Leaving a dead page live with noindex preserves an unnecessary URL and gives users no useful outcome.

Crawl-budget optimisation

Noindex is not a shortcut for stopping Google from crawling URLs. Google must request the URL to discover the directive, and its current crawl-budget documentation specifically advises large sites not to use noindex merely to save crawling time. Most small and medium sites do not need advanced crawl-budget work at all. For genuinely large URL inventories, consolidate duplicates, remove dead URLs correctly, keep sitemaps clean, and use robots.txt only where preventing crawling is the actual goal. Read our robots.txt mistakes guide before applying broad blocks.

Controlling only the search snippet

If the page should remain indexed but you want to limit what Google can show in a result, noindex is too blunt. Robots directives such as nosnippet, max-snippet, or noimageindex address specific presentation controls without removing the whole page from search.

How to Implement Noindex Safely

1. Confirm the URL’s intended outcome

Record the exact URL or template, why it should remain accessible, why it should not appear in search, who owns the decision, and when it should be reviewed. Check whether a canonical, redirect, deletion, or access control would better match the business requirement.

2. Add a robots meta tag to an HTML page

Place the directive in the HTML <head>:

<meta name="robots" content="noindex">

Using name="robots" addresses search crawlers that support the rule. A Google-only instruction can use name="googlebot", but a general robots directive is usually clearer when the indexing decision applies across search engines.

3. Use an X-Robots-Tag for a PDF or other non-HTML resource

Send the directive in the resource’s HTTP response:

HTTP/1.1 200 OK
X-Robots-Tag: noindex

Apply the header to the actual file response, not merely to the HTML page that links to it. Check your content delivery network, storage service, web server, and application layer so that one system does not strip or overwrite the header.

4. Do not add nofollow automatically

nofollow is a separate instruction about links on the page. The default behaviour is to allow link discovery, so noindex, follow is normally redundant and noindex, nofollow should not be copied by habit. Add nofollow only when you deliberately want that separate behaviour.

5. Keep the URL crawlable

Do not disallow the URL in robots.txt when you need Google to obey noindex. If crawling is blocked, Google cannot read the meta tag or HTTP header, and the URL may remain or appear in results without a normal snippet.

6. Serve the intended directive without relying on JavaScript to reverse it

Prefer a server-rendered meta tag or an HTTP header. Google warns that when it encounters noindex in the original response, it may skip rendering and JavaScript execution. A page that ships as noindex and depends on JavaScript to remove the tag may therefore remain excluded. The safer rule is simple: if a page should be indexable, do not send noindex in the initial response.

7. Remove the URL from XML sitemaps

Google’s sitemap guidance says to include the URLs you want to see in search results, which normally means the canonical URLs you want Google to consider for search. Keeping a noindexed URL in the sitemap sends conflicting operational signals and encourages continued discovery. Internal links are a separate decision: retain a link if users genuinely need the page, but do not feature a noindexed utility page as though it were an important organic landing page.

How to Verify Noindex—and Troubleshoot a URL That Still Appears

Check what your server actually sends

  • For an HTML page, inspect the raw page source and confirm the robots meta tag is present in the final response.
  • For a file or header-based implementation, inspect the HTTP response and confirm X-Robots-Tag: noindex is returned on the exact URL.
  • Check every important URL variant, including HTTP/HTTPS, www/non-www, trailing slash, uppercase variants, and parameterised versions where they resolve separately.
  • Confirm the URL is not redirecting. Google processes the redirect target rather than using the redirecting page’s content, so inspect the final destination as well.

Check that Google is allowed to see it

Review robots.txt and any firewall, bot-protection, login, or geo restriction. A URL can be accessible in your browser while Googlebot receives a different response. Also make sure mobile and desktop versions do not send different robots directives.

Use Google Search Console for the specific URL

Use URL Inspection to compare Google’s indexed information with a live test of the current URL. Confirm that Google can fetch the page and that the rendered or fetched response contains noindex. The Page indexing report can help you monitor groups of URLs where Google detected a noindex directive.

Allow time for a recrawl

Adding noindex does not remove a result instantly. Google has to revisit the URL, and its documentation warns that this can take a long time for infrequently crawled pages. You can request a recrawl through URL Inspection, but a request does not guarantee immediate processing.

Use the Removals tool only for urgent temporary suppression

When a result needs to disappear quickly, Search Console’s Removals tool can temporarily hide it. You must still implement a permanent outcome—such as removing the content, protecting it, returning an appropriate status, or adding noindex—because the removal request itself expires.

Do not use a site: search as your only proof

A site: query can be a quick spot check, but Google’s documentation says the results are not exhaustive. Use URL Inspection for a specific page and Search Console reporting for broader monitoring.

Four Common Cases—and the Correct Action

1. A public thank-you page after a generic enquiry form

The page must remain available after form submission, but it offers no useful search landing experience. Add a robots meta noindex, remove the URL from the sitemap, and verify that the form still redirects users correctly. Do not include submitted personal information in a publicly accessible URL or page.

2. A PDF intended only for paying members

Noindex alone is not enough. Put the file behind authentication or serve it through an authorised download process. An X-Robots-Tag is suitable only when the PDF may remain public but you do not want it listed in search.

3. Two product URLs show substantially the same item

If both URLs need to exist but one should be the representative search version, use a canonical and make internal links, sitemap entries, and other signals point consistently to the preferred URL. Noindex would remove the duplicate rather than clearly request signal consolidation.

4. An old service page has been replaced

Redirect the old URL to the closest relevant replacement. If the service has ended and no suitable replacement exists, return 404 or 410. Keeping an obsolete page live with noindex is less useful to people and leaves unnecessary maintenance behind.

Roll Out Noindex Without Accidentally Hiding Valuable Pages

  1. Test one representative URL first. Confirm the directive appears in the raw response, Google can crawl it, and the page still works for users.
  2. Check the rule’s scope. A CMS setting, shared template, HTTP-header rule, or deployment variable can affect an entire section—or the whole site—when you intended to change one page.
  3. Protect priority pages. Explicitly check the homepage, main category or service pages, important products, and other organic landing pages for accidental noindex.
  4. Keep mobile and desktop directives consistent. Different robots settings across page variants can cause unexpected indexing outcomes.
  5. Maintain a noindex inventory. Record the URL pattern, reason, owner, implementation method, and review date so temporary decisions do not become permanent by accident.
  6. Monitor the result. Use Search Console and analytics to confirm only the intended URLs leave search and that important impressions and visits remain stable.

If you later want a page indexed again, remove every noindex source—including HTTP headers, CMS settings, and duplicate meta tags—keep the URL crawlable, add the canonical URL back to the sitemap where appropriate, and request a recrawl. Conflicting robots directives are combined conservatively, so one forgotten noindex can continue to block the page.

Noindex Checklist and Official Resources

Pre-publish checklist

  • The URL should remain publicly accessible but should not appear in search.
  • Noindex is not being used to protect confidential or paid content.
  • A canonical, redirect, 404, or 410 is not the more appropriate outcome.
  • The directive is present in the HTML source or the exact resource’s HTTP response.
  • The URL is not blocked by robots.txt when Google needs to read noindex.
  • No unintended nofollow, duplicate robots tag, CDN header, or CMS setting changes the result.
  • The URL has been removed from XML sitemaps.
  • Desktop and mobile responses are consistent.
  • The implementation has been checked in Google Search Console.
  • An owner and review date are recorded for temporary or large-scale rules.

Primary Google documentation

Noindex is a precise tool, not a general clean-up switch. Use it when a URL should stay available but should not be a search result; choose a canonical, redirect, deletion response, crawl control, or access restriction when that better describes the outcome you actually need.

Ready to Get More Out of ScanMySEO?

Whether you're just getting started or already have scans to review, take the next step towards boosting your search presence.

Register for Free
Hansel McKoy

Hey there, I'm Hansel, the founder of ScanMySEO. I've spent over ten years helping global brands boost their digital presence through technical SEO and growth marketing. With ScanMySEO, I've made it easy for anyone to perform powerful, AI-driven SEO audits and get actionable insights quickly. I'm passionate about making SEO accessible and effective for everyone. Thanks for checking out this article!

Hansel McKoy

Founder, ScanMySEO


Get More Out of ScanMySEO