AI Search Indexing: How ChatGPT Search and Perplexity Discover Pages
When an AI answer cites a page, three things happened first: the page was discovered, it was fetched, and it was stored somewhere an engine could retrieve it from. That is indexing, and it did not stop mattering when the results page turned into a paragraph. AI search indexing is the unglamorous layer underneath ChatGPT Search, Perplexity, and every other answer engine. The layer works differently for each of them, and the differences decide whether your pages can be cited at all. This post covers how the two most-used non-Google answer engines actually discover pages, and what that means for the work you already do.
Three ways an answer engine can know about a page
Strip away the branding and there are only three routes.
Its own crawler. The engine runs a bot, the bot follows links and sitemaps, and the engine builds an index it controls. This is the classic model, and it is how Google indexing has worked for decades: discovery, crawl, then a decision about whether the page is stored.
Someone else’s index. Building a web-scale index is expensive, so younger engines license one. An engine that draws on an established search index inherits its coverage: pages that index knows about are visible, pages it missed are not, and no amount of answer-engine cleverness recovers a page the underlying index never stored.
A live fetch at question time. When a user asks about a specific URL, or the engine decides its stored copy is stale, it fetches the page on the spot. This route does not require the page to be indexed anywhere, but it only triggers when something points the engine at the URL. It is a retrieval of last resort, not a distribution channel.
Every answer engine mixes these three routes in different proportions. The mix is what you need to understand.
How ChatGPT Search discovers pages
OpenAI documents separate user agents for separate jobs. GPTBot collects content for model training. OAI-SearchBot crawls for the search product. ChatGPT-User fetches a page when a user’s request points at it directly. Each can be allowed or disallowed independently in robots.txt, which matters, because the common instinct to block GPTBot (training) also removes you from search results if you block the whole family without reading the names.
OpenAI’s search product has also drawn on third-party search indexes alongside its own crawling. The practical consequence: your visibility in ChatGPT Search is not a single switch. It depends on whether OAI-SearchBot can reach you, and on whether the conventional search indexes it supplements from have your pages. Bing accepts submissions through the IndexNow protocol, and an engine leaning on Bing’s index inherits whatever IndexNow-submitted pages Bing chose to keep. The comparison between that route and Google’s is a separate topic, covered in IndexNow vs Google’s Indexing API.
How Perplexity discovers pages
Perplexity runs PerplexityBot to build its own index, and uses a separate user-initiated agent for live fetches when someone asks about a specific page. The model is closer to a conventional search engine than most people assume: crawl, store, retrieve, then generate the answer on top of the retrieved set.
Which means the conventional failure modes apply. A page PerplexityBot cannot reach is a page Perplexity answers cannot cite. A site whose robots.txt blocks unfamiliar bots by default, a common posture on staging configurations that quietly ship to production, is invisible to it. Google’s own documentation on robots.txt describes the mechanism, and the mechanism is engine-agnostic: every reputable crawler reads the same file and obeys the same directives.
AI search indexing rests on ordinary indexing
Here is the uncomfortable summary. There is no separate submission pipeline for answer engines. No form where you register a page for citation. The routes in are the same routes that have always existed: be crawlable, be linked, be in a sitemap, be in the indexes these engines build or borrow. Google publishes a crawlers overview listing its own user agents for the same reason OpenAI and Perplexity publish theirs: the crawl layer is the contract, and robots.txt is how you sign it.
So the checklist for AI search visibility is mostly the checklist you already had. The things that make a page hard to index for Google, a stray noindex, a robots block, a canonical pointing elsewhere, thin content, make it hard to index for everyone. Fixing them pays out across every engine at once.
Where Google’s AI surfaces fit
Google’s AI Overviews and AI Mode sit on top of Google Search itself, and they draw on Google’s index. Google’s documentation on AI features in Search is explicit that standard indexing controls apply. A page absent from Google’s index cannot appear there, whatever its merits. How those surfaces select and cite pages is its own subject, covered in Google AI Mode: what it is and how pages get cited.
This is also the one part of the stack you can actually verify. You cannot query Perplexity’s index, and OpenAI offers no coverage report. Google’s index you can check directly, and it is the base layer Google’s own AI surfaces draw from. That is the check AIndexify runs: submit a URL, and the tool checks it against live Google results on a schedule, reporting submission and indexing as two separate events, because delivery succeeding and Google keeping the page are different facts.
What to do this quarter
Read your robots.txt and name the bots you are blocking, deliberately or by wildcard. Decide, per crawler, whether you want that engine citing you. Keep your sitemap current, since every crawler on this page reads it. Then verify the one index you can verify, instead of assuming.
See where your pages actually stand on the one index you can verify: submit your URLs and get the indexed-or-not verdict per URL, with the date it was found.