Six reasons Google declines to index a page that looks fine
Submitting a URL asks Google to look. It does not oblige Google to keep what it finds. When a page is submitted cleanly and still does not appear, the cause is almost always one of six things, and they are distinguishable.
1. The page tells Google not to
A noindex meta tag or X-Robots-Tag header is an instruction, and Google follows it.
This is the most common cause and the easiest to miss, because the page looks completely
normal to a human. Staging locks left on after launch account for a surprising share of
these.
2. robots.txt blocks the crawl
A disallowed path cannot be fetched, so there is nothing to index. Worth knowing: this
is different from noindex. A blocked URL can still appear in results as a bare link if
Google finds enough external signals, which is usually the worst of both worlds.
3. Google chose a different canonical
If the page is a near-duplicate of another, Google may fold it into that one. The page is not rejected; it is consolidated. You will see the other URL ranking instead.
4. The content is thin relative to what already ranks
Indexing is not free for Google either. A page that adds nothing over pages already in the index is a poor use of their storage, and they act accordingly.
5. The site has no crawl history
A new domain with no inbound links gets crawled sparsely. Submission helps here more than anywhere else, but it is a nudge rather than a guarantee.
6. It simply has not happened yet
Indexing is not instantaneous, and “not yet” looks identical to “never” until enough time has passed. This is exactly why we check on a schedule rather than once, and why we distinguish not yet indexed from not indexed as of a date.
Telling them apart
The first two are detectable before a check is even billed, which is why our preflight reports them as Couldn’t check with the specific reason rather than spending a credit to discover the obvious. The remaining four need time and repeated checks, which is what the scheduled checks are for.