robots.txt Disallow: The Indexing Trap Most Sites Fall Into

A robots.txt Disallow rule looks like an indexing control. It is not one. It is a crawling control, and the difference between those two words is responsible for two of the most common indexing problems on the web: pages that stay indexed after you tried to remove them, and pages that vanish from Google because of a line someone forgot to delete.

Google is plain about this in its own documentation: robots.txt is a way to manage crawler traffic to your site, and it is not a mechanism for keeping a web page out of Google. Most sites read the first half of that sentence and skip the second. This guide covers what Disallow actually does, the two traps it sets, and how to check whether one of them has already caught you.

What Disallow actually does

A robots.txt file sits at the root of your host and tells crawlers which paths they may request. A Disallow rule for Googlebot means Google will not fetch matching URLs. That is the entire contract. Nothing in it says “and remove these URLs from the index” or “and never show them in results”.

User-agent: *
Disallow: /drafts/

This tells every compliant crawler not to request anything under /drafts/. Google will honour it. What Google will not do is forget that /drafts/ exists, because crawling and indexing are separate stages, and Disallow only touches the first one.

Trap one: blocked but indexed anyway

Here is the part that surprises people. Google can index a URL it has never crawled.

If other pages link to a disallowed URL, Google knows the URL exists, knows the anchor text pointing at it, and can decide the URL belongs in the index on that basis alone. The result appears in Search Console as “Indexed, though blocked by robots.txt”, and in search results as a bare URL with no description, because Google was never allowed to read the page and write one.

This is why using Disallow to hide a page is the wrong tool. Google’s guidance for keeping a page out of results is a noindex rule, delivered in the page’s meta tags or its HTTP response headers. A noindex is an instruction about the index. A Disallow is an instruction about crawling. They answer different questions.

Trap two: Disallow cancels your noindex

The second trap is crueller because it catches people who did read the documentation.

A noindex tag only works if Google fetches the page and sees it. If the same URL is also disallowed in robots.txt, Google never requests the page, never encounters the noindex, and the instruction might as well not exist. Google’s own documentation states this directly: for noindex to be effective, the page must not be blocked by a robots.txt file.

So the combination that feels safest, Disallow plus noindex, is the one configuration in which the noindex can never be seen. If the page picks up links, you can end up with the worst of both: an indexed URL you cannot describe and cannot deindex, because the removal instruction is sitting behind a door you locked yourself.

If you want a page out of the index, the order of operations matters. Let Google crawl it, let Google see the noindex, and only consider a Disallow after the page has dropped out, if you still want to save the crawl at all.

The accidental Disallow: how sites block themselves

The traps above are about pages you wanted hidden. The more expensive version involves pages you wanted found.

The staging file that shipped. A development site often carries Disallow: / to keep unfinished work out of Google. When the site launches, the file sometimes launches with it. One character after the slash decides whether your site is open or closed, and nothing on the rendered page looks any different.

The over-broad path. Disallow: /search was meant to block internal search results, but it also matches /search-engine-guide/. Rules match by prefix unless you end them with $, and a rule written for one template quietly swallows another.

Case and subtlety. Rules are case sensitive: Disallow: /PDF/ does not block /pdf/. Wildcards, ordering, and the interplay of Allow and Disallow rules have precise semantics that are easy to half-remember. Google’s page on creating a robots.txt file documents the rules; most robots files were written from memory instead.

Blocked resources. Disallowing your CSS or JavaScript directories does not block your pages, but it can stop Google rendering them properly, which affects how they are evaluated. The rule looks harmless because the blocked URLs are not pages anyone searches for.

A robots.txt problem is one of several reasons a healthy-looking page stays out of the index; we walk through the full list in six reasons Google declines to index a page that looks fine.

How to find out whether this is your problem

Start with the file itself: fetch yoursite.com/robots.txt in a browser and read it line by line, against the actual URL paths on your site rather than the paths you remember having. Then test specific URLs. Search Console’s URL Inspection tool will tell you whether a URL is blocked by robots.txt and whether it is indexed, which covers both traps in one report.

The distinction to keep in mind while you check is the one this whole subject turns on: whether Google fetched a page and whether Google indexed it are two separate events, and no single green tick covers both. A page can be crawled and still sit unindexed, a state with its own causes and fixes, and a page can be indexed without ever being crawled, as trap one shows.

Checking at scale is duller. Reading one robots file is easy; verifying that 3,000 product URLs are actually fetchable and actually indexed is not a browser job. This is one of the places a submission tool should refuse to guess: when AIndexify runs its preflight on a submitted URL and finds it blocked by robots.txt, it reports “Couldn’t check”, refunds the credit, and bills nothing, because submitting a URL that Google is forbidden to fetch is not a service, it is a fee.

The fix, in order

1. Decide what you actually want. Out of the index means noindex, with the URL crawlable. Less crawling of unimportant paths means Disallow. The same URL should almost never need both at once.
2. Repair the file. Remove rules that block pages you want indexed. Narrow over-broad prefixes with more specific paths or $ anchors.
3. Unblock before you deindex. If a disallowed URL is stuck in the index, lift the Disallow first so Google can reach the noindex you add.
4. Recheck on a schedule. Robots files change when platforms update, plugins install, and staging configurations ship. A rule that was correct in March can be wrong in May, and nothing will notify you.

robots.txt is a good tool for the job it was built for: steering crawlers away from the parts of a site that waste their time. It was never built to manage what Google keeps. If you have fixed a Disallow and want evidence of what Google now does with those pages, submit your URLs and watch submitted and indexed reported as the separate events they are.