Crawling, indexing, serving

Search is not a search of the web. It is a search of a copy of the web that Google built earlier. Almost every confusing SEO situation becomes obvious once you hold that distinction: the thing being ranked is Google's stored understanding of your page, not your page.

That copy is built in three stages.

Crawling

Googlebot fetches pages. There is no registry of all URLs, so Google has to discover them — mostly by following links, plus sitemaps you submit. Discovery and fetching are separate from everything else: a page can be known about for weeks without being fetched, and fetched without being stored.

Crawling is rate-limited by what your server can take and by how much Google thinks the site is worth fetching. That budget is rarely the problem on a small site and often the problem on a large one.

Indexing

Google analyses what it fetched — text, images, video, structured data — works out what the page is about, and decides whether to store it. Indexing is a decision, not an automatic consequence of crawling. "Discovered – currently not indexed" and "Crawled – currently not indexed" in Search Console are Google saying it looked and declined.

Two things surprise people here. First, pages that need JavaScript to render their content go into a render queue and are processed later, so their content lands in the index after their HTML does. Second, Google uses the mobile version of your page for indexing and ranking — if content only exists in the desktop layout, it effectively does not exist.

Serving

When someone searches, Google selects from the index and orders the results. Nothing is computed against the live web at this moment. If you fixed a page an hour ago, the fix matters only once it has been re-crawled and re-indexed.

What trips people up

• "I published it, why isn't it ranking?" Usually it is not indexed yet. Check indexing status before touching content.
• Confusing crawl blocking with index blocking. robots.txt stops fetching; noindex stops storing. Blocking a page in robots.txt means Google cannot read the noindex on it — the two do not stack, they conflict.
• Expecting instant effect. Every change is subject to the round trip: crawl, index, then serve.

You have got this when

You can explain why a page that returns a 200, is linked from your homepage, and has noindex in its HTML will never rank — and why removing the noindex still will not make it rank today.