One-Thread Crawls of 30-50 URLs per Locale Cut Audit Volume Below 403 WAF Limits
Most mid-size sites now sit behind Cloudflare, Akamai, or DataDome, which score a crawler's TLS fingerprint and request pattern rather than just its IP and UA.
Overwhelming bot traffic drives this lockdown: before dialing their WAF to 11, one webmaster paid 3,000 EUR in infrastructure costs from a single night of bot hits.
Hitting a 403 status code after ~50 requests is usually a rate-limit rule trigger.
While one specialist still usually switches the crawler UA to Googlebot and drops simultaneous threads to 2, modern WAF setups verify Googlebot against Google's IP ranges and reverse DNS to catch spoofed bots.
In server logs another specialist checked in October 2026, over 1 in 4 "Googlebot" hits were fake.
A four-step sampling workflow keeps international audits moving without tripping those rate rules:
- Extract
hreflangannotations fromXMLsitemaps first, which often removes the need to crawl. - Run 1-thread crawls of 30-50 URLs per locale with a 2-3 second delay to cut audit volume below
403WAFlimits instead of crawling the full site. - Check page rendering on a few templates inside a real browser.
- Request an
IPor header allowlist — or a whitelisted crawl window — when auditing a client site.
Geo-located IP addresses and browser rendering are also unnecessary for auditing hreflang.
Because the annotations sit in the HTML <head>, the HTTP header, or the XML sitemap, one plain fetch per URL shows what Google sees.
Furthermore, Googlebot crawls mostly from US addresses; if a site only serves its Japanese or German version to local IP addresses, the crawler never reaches those pages, making that geo-restriction an audit finding in itself under Google's guidance against redirecting by location.
Residential proxies can bypass WAF restrictions on competitor domains, but they route traffic through home connections to evade rules the target site configured on purpose, and the device owners usually do not know their connection is being used or may have consented accidentally.