The Indexing API for Bulk Pages: A Practical Guide

A developer holding forty thousand URLs and a deadline finds the Indexing API in about ten minutes, and it reads like the answer: a documented Google endpoint, a service account, one POST per URL, no waiting for a crawler to wander past. The integration itself is genuinely simple. The complication sits upstream of the code, in what Google says the endpoint is for, and it decides whether writing the rest is worth the time. What follows is the mechanics you would ship, the limits a bulk set hits, and the honest options for everything the API will not take.

What the Indexing API actually accepts

Google’s Indexing API quickstart states the supported content in its first lines: pages carrying either JobPosting structured data or a BroadcastEvent embedded in a VideoObject. That is the documented scope, and both types go stale fast. A vacancy closes, a livestream ends, and a result pointing at either is worse than no result. The API exists so those pages can be picked up and dropped again quickly.

The endpoint does not inspect your markup when the request arrives, which is why plenty of scripts push ordinary product and blog URLs through it and sometimes watch a crawl follow. Be clear-eyed about what that rests on: behaviour Google documents as out of scope, against a quota provisioned for a different job, and the quickstart notes that attempts to abuse the API, including using multiple accounts to exceed usage quotas, may result in access being revoked.

So if your bulk set is job listings or livestream pages, the API is the right channel and the rest of this guide is your build. If it is not, the last two sections still apply to every route.

What a working integration looks like

Authorisation is where most first attempts stall, and it has nothing to do with API keys.

1. Enable the Indexing API in a Google Cloud project and create a service account inside that project. Download its JSON key and treat it as a credential, not as config: out of the repository, into your secret store.
2. Add the service account as an owner of the property in Search Console. It has an email address of its own, and that address needs owner permission on the property containing your URLs. This is the entire authorisation model: no page tag, no extra header, but the property must be verified and a lesser permission level will not do.
3. Fetch an access token for the https://www.googleapis.com/auth/indexing scope using the service account key. Google’s client libraries do this for you, and the token is short-lived, so cache and refresh rather than minting one per URL.

The calls themselves are small. A POST to urlNotifications:publish carries a body with two fields, the url and a type of either URL_UPDATED or URL_DELETED. Updated means new or changed. Deleted is a request to drop the page, not a way to force a fresh look, so send it only when the URL genuinely returns a 404 or a 410. A companion method, getMetadata, returns the last time Google received each kind of notification for a URL.

Two things belong in the code from the start. Send nothing you have not checked, since a URL that 404s, redirects, or sits behind a Disallow rule spends a quota unit to tell Google something unhelpful. And log the full response per URL rather than a success count, because it is the only record of what was accepted and when.

Batching reduces connections, not quota

This is the most expensive misreading in bulk work. The quickstart describes combining up to 100 calls into a single HTTP request, and separately mentions a default quota of 200 provided for onboarding and submission testing, with more available on request.

Those two numbers measure different things. Batching is a transport convenience: 100 calls travel over one connection instead of a hundred, which saves latency and sockets. Every call inside the batch still counts as a call. A hundred-URL batch does not cost one unit, it costs a hundred, and forty thousand URLs are not going anywhere on a default onboarding quota however neatly they are packed.

Build for that. Bulk submission here is a queue with a rate limiter, not a loop over an array: a worker that respects the quota, backs off exponentially on rate-limit responses, retries transient failures a bounded number of times, and marks the rest as delivery failures to inspect. Resist re-notifying the same URL on a timer, too. Google’s guidance on asking for a recrawl is blunt about the equivalent behaviour elsewhere: requesting a recrawl repeatedly for the same URL will not get it crawled any faster, and crawling can take anywhere from a few days to a few weeks.

The pages the API will not take

For an ordinary catalogue, a documentation set, or a programmatic build, the documented routes are the unglamorous ones, and a sitemap is still the discovery mechanism designed for volume. One file holds up to 50,000 URLs or 50MB uncompressed, and larger sets split across several files behind a sitemap index. Keep the last-modified dates truthful, because a file that claims everything changed last night teaches Google to discount the field. Note the same page’s caveat: submitting a sitemap is a hint, and it does not oblige Google to download the file or crawl what it lists.

Before any of that, spend an hour on eligibility, because submission of any kind is wasted on a page that cannot be indexed. The reasons Google declines a page that looks fine are mostly mechanical: a stray noindex, a robots rule covering the directory, a canonical pointing somewhere else, a route that renders empty without JavaScript. Fixing one of those in a template fixes it across every URL the template produces.

What the API tells you, and what it does not

getMetadata answers one question: when Google last received a notification for this URL. That is receipt, not judgement. It says nothing about whether the page was crawled, and nothing about whether it was kept, which is the gap between submitted and indexed that quietly ruins bulk reporting. A pipeline can show forty thousand accepted notifications while the index holds nine thousand of those pages, and every number on that dashboard is technically true.

So the verification half is yours to build, whichever channel you use. That is the job AIndexify does: URLs arrive pasted or as a CSV, up to 10,000 per submission, with no Search Console connection or ownership proof required, and each is then checked against live Google results on a fixed schedule. Pages carrying a noindex, a 404, or a robots block are caught before a check is billed and the credit is refunded, and the result is a date the page was found or a settled “not indexed as of” answer rather than an open-ended pending state. If you would rather write that yourself, how Google reaches those decisions is laid out in our guide to Google indexing.

Whichever way your URLs reach Google, the checking is the part that tells you something. Submit your URLs and see, with dates, which of them Google actually kept.