The X-Robots-Tag Header: The noindex You Didn't See

There is a version of noindex that never appears in your HTML. The X-Robots-Tag header travels in the HTTP response, one layer above the page itself, and it carries exactly the same authority as the robots meta tag everyone knows to check for. When a page refuses to index, the usual routine is to view the source, find no noindex, and move on to theories about content quality. Meanwhile the header sits in the response, invisible to a source view, quietly telling Google not to keep the page.

This post covers what the X-Robots-Tag header does, where it tends to hide, and the small number of cases where it is genuinely the right tool.

What the X-Robots-Tag header does

Google’s documentation on blocking indexing with noindex is plain about it: any rule that works in a robots meta tag can also be sent as an X-Robots-Tag header in the HTTP response. A response carrying it looks like this:

HTTP/1.1 200 OK
Content-Type: text/html
X-Robots-Tag: noindex

To Google, that response is equivalent to a page with <meta name="robots" content="noindex"> in its head. The page can be crawled, but it will not be kept in the index, and if it was indexed before, it will be dropped once Google recrawls it.

Two properties separate the header from the meta tag. First, it works on files that cannot carry a meta tag at all: PDFs, images, video files, anything that is not HTML. Second, it is set in server or edge configuration rather than in a template, which means a single rule can cover thousands of URLs at once. That is its main strength when used deliberately, and its main danger when nobody remembers it is there.

Why you didn’t see it

Viewing source shows you the document. It does not show you the response headers that delivered the document, and neither does most CMS tooling. That is the whole reason this particular noindex goes unnoticed for months: the place people look and the place the directive lives are different layers.

The header usually arrives from one of four places:

• Server configuration. An Apache .htaccess rule or an nginx block adds the header for a path or a file type. These rules are often written once, for a good reason, and never revisited.
• CDN or edge rules. A header injected at the CDN can apply more broadly than the person who wrote it intended, and it will not appear anywhere in your origin’s code.
• Staging settings that survived launch. Platforms and deploy setups commonly add X-Robots-Tag: noindex to non-production environments. When the same configuration ships to production, the protection ships with it.
• Plugins and middleware. Security and SEO plugins can set response headers. Two plugins with overlapping opinions is a classic source of a header nobody chose.

How to find the X-Robots-Tag header

You need to look at the response, not the page. One line does it:

curl -I https://example.com/page

Read the output for an X-Robots-Tag line. The Network tab of your browser’s developer tools shows the same thing under response headers. For confirmation from Google’s side, the URL Inspection tool reports whether indexing is allowed for the URL, whichever layer the block came from.

Checking one URL is quick. Checking a section of a site is where this gets tedious, because a path-level header means every URL under that path is affected. This is one of the checks AIndexify runs before it bills anything: each submitted URL gets a preflight, and a noindex, a robots block, or a 404 comes back as Couldn’t check, with the credit refunded and no check billed. A header you could not see in your HTML becomes a visible verdict in the list, which is a faster way to map the damage than curling URLs one at a time.

When the header is the right tool

None of this makes X-Robots-Tag a mistake in itself. It is the correct choice in a few situations:

• Non-HTML files. A PDF cannot carry a meta tag. If you need it out of the index, the header is the way.
• Rules at path scale. Keeping an entire /internal/ or /search/ tree out of the index is one server rule instead of a template change across systems.
• Responses you do not template. API output, generated exports, printer views.

For ordinary pages, the meta tag is easier to see, easier to audit, and closer to the people who edit content. The decision of when noindex is helping you and when it is quietly costing you traffic is the same for both forms, and we cover it in the noindex tag guide.

The robots.txt trap

One interaction catches people out repeatedly. For any noindex to work, Google has to fetch the URL and see the directive. If robots.txt disallows the URL, Google never makes the request, never sees the header, and the page can remain indexed from links alone. Google’s documentation states this dependency explicitly. Blocking crawl and blocking indexing are different controls, and combining them badly gets you the worst of both; the robots.txt disallow trap walks through that failure mode.

The reverse cleanup has its own lag. Removing a stray X-Robots-Tag does not restore the page; Google has to recrawl and reconsider it, on Google’s schedule. Where that step fits in the wider picture, from discovery through crawling to the index, is laid out in the ultimate guide to Google indexing.

If a set of pages has been refusing to index and the HTML looks clean, the response headers are worth a look, and so is the record. Submit your URLs and see where they actually stand.