When robots.txt and the edge disagree, the crawler loses
A services site in the measured fleet permits Claude-SearchBot in robots.txt. Its edge rules have since refused 1,999 of that crawler’s requests with 401 or 403.
The stated policy says welcome; the firewall says no. Since 2026-07-31, one services property has refused 1,999 requests from Claude-SearchBot with 401 or 403 responses while its own robots.txt explicitly allows that crawler — 99% of its non-probe requests went unanswered. The same property refused another 52 requests from GPTBot against an identical allow.
Crawlers do not read intent; they read status codes. A contradiction between robots.txt and edge rules presents as a hard refusal, and no policy document fixes what the firewall rejects.
The audit pattern: reconcile WAF and edge rules with the stated robots policy, allowlist the AI user agents that should reach content, then verify against the same request-level measurement that found the contradiction. Numbers as of 2026-08-07, probes excluded.