How to fix a robots meta tag that blocks indexing

Remove the noindex value at whatever is producing it — the CMS privacy setting, the page template, or the server's X-Robots-Tag header — then let Google recrawl the page. A leftover <meta name="robots" content="noindex">, usually inherited from a staging environment, silently removes pages from search with no error anywhere. Always audit it across the whole site, because a single shared template can noindex hundreds of URLs at once.

Last updated 2026-07-28

What does the robots meta tag control?

It controls what a search engine may do with a page it has already fetched: whether the page can be indexed, whether its links pass signals, and how much of it can be shown in a result. It has no effect on whether the page gets crawled — by the time the tag is read, the crawl has happened.

That distinction is the whole point of the tag. It is an instruction about the index, delivered inside a page that the crawler was allowed to read.

The default is index, follow. You do not need the tag to get that behaviour, which means any robots meta tag on your site exists to restrict something, and every one of them deserves a reason.

What is the difference between noindex and robots.txt?

robots.txt blocks crawling; the robots meta tag blocks indexing. A URL disallowed in robots.txt is never fetched, but it can still appear in results as a bare link with no snippet, because Google knows it exists from inbound links. A URL with noindex is fetched, read, and then deliberately kept out of the index.

This produces the trap that catches people most often: if you block a URL in robots.txt, Google can never read the noindex tag on it, so the page keeps lingering in search results and no amount of waiting fixes it. To remove a page properly you must allow crawling and serve noindex.

It is also worth knowing that noindex as a robots.txt directive has not been supported since September 2019, when Google formally dropped the undocumented behaviour. Any noindex lines still sitting in a robots.txt file today are doing nothing at all.

Which robots directives actually matter?

In practice five do: noindex, nofollow, noarchive, and the two preview limiters max-snippet and max-image-preview. Everything else in the specification is either the default or so situational you will know when you need it.

Where do stray noindex tags come from?

Four sources account for nearly all of them, and every one is an inheritance rather than a decision someone made about that page. Nobody types noindex onto a page they want ranked — it arrives from a setting, a template, or an environment.

• A staging environment promoted to production with its blocking tag still in place. This is the most common single cause and the most damaging, because it lands on every URL at once.
• A CMS privacy setting left on — WordPress's "Discourage search engines from indexing this site" checkbox is the classic case, and it is the first thing to check on any WordPress audit.
• A template applied to a section, such as a blog archive or a tag listing, that quietly became the base template for everything else.
• An X-Robots-Tag HTTP response header set at the server or CDN, which never appears in page source and so survives every visual inspection.
• An SEO plugin rule that noindexes a post type or taxonomy by default, applied before anyone checked which URLs it would catch.

Why is my page still in Google after I added noindex?

Because Google has not recrawled it yet, or cannot. The tag only takes effect when the crawler fetches the page and reads it, so there is always a delay between deploying the fix and the page disappearing — days for a frequently crawled URL, weeks for one that is rarely visited.

If the page never disappears, the usual cause is that the same URL is also disallowed in robots.txt. The crawler is blocked from fetching it, so it never sees the noindex, and the URL can persist indefinitely as a title-less entry. Unblock crawling and the directive will finally be processed.

To speed things up, request indexing for the URL in Search Console — the same tool works for removals as well as additions. For an urgent removal, the Removals tool hides a URL from results within hours while the noindex works through the normal crawl cycle.

How do I check the robots directive on every URL?

Crawl the site and record the robots directive per URL, from both the meta tag and the X-Robots-Tag header, then look at the pattern rather than the individual rows. A template-level noindex appears as a contiguous block of pages sharing the same directive, which is far easier to spot than one anomalous page in a list of hundreds.

Audra records the robots directive for every crawled page and reports them together, so that block is visible at a glance rather than something you would have to notice. Because the crawl runs locally against your own installed Chrome, you can point it at a staging site before launch — which is the cheapest possible moment to catch a staging noindex.

While you are there, check the pages that should be noindexed and are not. Utility URLs like internal search results and filtered listings usually belong out of the index, and they overlap heavily with the thin pages worth consolidating.