Robots.txt Disallow Rules Kill Canonical Consolidation on Parameter URLs

Blocking parameter-based duplicate URLs via robots.txt blinds crawlers to canonical tags, warns Jono Alderson.

When SEOs attempt to clean up index pollution by adding a disallow rule for generated query parameters, they inadvertently trap those exact URLs in the search results.

Canonical tags live inside the page HTML.

Because the robots.txt directive triggers before a crawl, it blocks Googlebot from accessing the URL.

The crawler never renders the page to discover the <link rel="canonical"> tag, failing to consolidate the duplicate signals back to the primary product page.

This forces Google Search Console to flag the URLs as Indexed, though blocked by robots.txt.

To execute proper deduplication, strip the disallow rules and allow the crawler to process the canonical directives directly.