200-Domain Experiment Suggests Google and Bing Filter Common Crawl Data for LLMs

Common Crawl allocates crawl priority using monthly domain- and host-level harmonic centrality rather than PageRank, explains Stephen Burns.

The algorithm scores proximity based on the shortest-path graph distance, representing the fewest hops to the web's central core nodes.

Securing backlinks from these central nodes accelerates CCBot crawl frequency and expands ingestion into pre-training corpora like Google's C4.

This mechanism directly embeds entity and brand associations into the foundational parametric memory weights of an LLM rather than optimizing for real-time retrieval-augmented generation.

Separately, clearing ingestion into the 2.3-billion-page monthly corpus demands passing three technical gates, details Stephen Burns.

First, explicitly allow CCBot in the robots.txt file.

Second, whitelist the bot against preset AI-blocking firewall filters at the CDN and edge layers, specifically on Cloudflare and Akamai.

Third, serve content in raw server-side HTML and text.

The crawler ignores JavaScript, images, and video entirely, rendering client-side sites invisible in the dataset.

The bot runs a discovery phase utilizing harmonic centrality, where a single link from a high-centrality domain can outperform 50 high-PageRank links for triggering inclusion.

It then executes a multi-week crawl compiling 120TB of web data released publicly at the end of each month.

Conversely, regarding system validation, I deploy a 200-domain controlled experiment blocking all bots except CCBot to isolate ingestion behaviors, reveals James Dooley.

The test featured synthetic entities with links built strictly 1 tier deep from CC Rank 1 trusted seed sites via harmonic centrality.

After 10 weeks, the setup produced zero LLM entity recognition.

The data suggests LLMs do not directly ingest raw Common Crawl datasets into active retrieval corpora without intermediary validation from Google or Bing indices acting as spam and precision filters.

Furthermore, for real-time query fan-out retrieval, scale off-page consensus by compiling verified review snippets into multi-format assets.

Distribute PDFs to SlideShare, publish image blasts with embedded text descriptions, syndicate video and audio across 47 podcast directories, and seed review quotes onto Reddit with force-indexing to generate cheap corroboration nodes.