Google Search Central Disclosures Map Six Gates Across the 20-Hour Crawl-to-Index Pipeline

Why do newly published URLs sit in "Discovered – currently not indexed" or wait weeks for a first crawl while other domains enter the index within hours?

Sören Bendig reports that internal timelines shared by Gary Illyes at Google Search Central Live in Barcelona set the average benchmark at 20 hours for Google to discover a new URL and 1.5 hours to index a page once processed.

Google speakers note they are roughly 90% certain on individual subsystem behaviors due to infrastructure complexity, framing these timelines as general averages.

When a domain suffers from technical or quality issues, it misses its scheduled processing time slot, stretching new URL discovery: if Google takes 3 to 4 weeks to find a new one instead of the normal 20 hours, that says something about the domain or project.

Separately, Tobias Schwarz describes how Google runs Machine Learning models prior to crawling to estimate whether a URL will yield added value before fetching any HTML.

While Google assigns a domain-wide quality classification that grants high-quality sites much wider leeway during new URL discovery, the pre-crawl ML gate halts low-value URLs upfront and surfaces them as "Discovered – currently not indexed" in GSC.

Without reading page content, the model evaluates URL quantity thresholds, groups URLs by pattern clusters, and presumably flags redundant variants caused by faulty HTTP-to-HTTPS migrations, relative links, or reordered query parameters.

High volumes in this report indicate either a crawl budget bottleneck or underlying URL-generation flaws that require pruning.

Furthermore, once crawling begins, Googlebot evaluates 500 and 429 server errors in 4-hour buckets.

Google throttles crawl velocity only when error rates show a significant change within a 4-hour window.

If a domain returns a constant, established baseline of 500 errors, the system recognizes that baseline and ignores the errors without slowing crawl speed.

Additionally, Google tries to filter out spam, duplicates, and low-value pages before index ingestion rather than suppressing them later at the ranking stage.

A "Crawled – currently not indexed" status in GSC means Google fetched the page, but its Index Selection ML systems rejected the document.

The rejection triggers when the system classifies the page as Commodity Content — non-unique material already mirrored across 1,000 other sites — or identifies it as a duplicate, near-duplicate, or Soft-404.

Regarding rendering, a noindex directive placed in the raw HTML causes Googlebot to skip JavaScript execution and page rendering.

Client-side JavaScript execution cannot revert an HTML-level noindex tag back to index, so Google aborts rendering as irrelevant.

Any secondary navigation added through rendering on that page remains unseen by Google.

Separately, during the indexing phase, Google runs positional detection to identify page regions — such as Main Content, Header, or Footer — and annotates extracted text with its page location.

The system uses these positional annotations for SERP presentation and ranking weight, discounting text placed far down the page while assigning higher weight when core copy moves higher up, ideally above the fold.

The same positional rule governs video indexing: embedded videos must have a dedicated view page and sit above the fold, or Google will not recognize and surface them as standalone videos.