HTTP 418 ModSecurity Blocks Against AI Crawler Retrieval: An Evidence-Based Investigation of Origin Security and Machine Accessibility

This technical report documents an investigation into AI crawler retrieval failures caused by ModSecurity at a website's origin server, despite the website appearing fully operational to human visitors. Sydney Business Web's AI Observatory identified abnormal retrieval behaviour affecting ClaudeBot. During the observed evidence window, only 1 of 7 business retrieval attempts succeeded, representing a 14.3% success rate. Controlled requests reproduced an HTTP 418 response. Comparative testing through Cloudflare and directly against the hosting origin isolated the rejection to the server-side security layer rather than the Cloudflare edge. Following corrective action by the hosting provider, the same controlled request returned HTTP 200 OK, confirming successful retrieval in the retest. The report documents a reproducible engineering methodology: Detect abnormal crawler retrieval behaviour. Reproduce the HTTP failure independently. Isolate the responsible infrastructure layer. Apply targeted remediation. Verify successful retrieval after correction. This investigation demonstrates why conventional website uptime and human browsing tests do not necessarily establish machine accessibility. It also distinguishes verified crawler retrieval from broader claims about AI visibility, recommendations or search rankings. Original technical investigation: https://sydneybusinessweb.com.au/ai-crawler-blocked-by-modsecurity/ Author: Keith Rowley, Sydney Business Web, Australia. Version: 1.0 — November 2026.