Unequal access to popular domain home pages across declared user-agent labels: A large-scale paired audit

This study examines whether declared user-agent labels are associated with differences in access to popular domain home pages and in representations after access succeeds. A paired cross-sectional audit requested the HTTPS home pages of 20,000 Tranco-ranked domains from one fixed research host using Chrome, Googlebot, OAI-SearchBot, GPTBot and ChatGPT-User labels. The Chrome label received HTTP 2xx responses from 60.10 percent of domains, compared with 54.97 percent for Googlebot and between 49.34 and 51.34 percent for the OpenAI labels. Most dual-success pairs had matching extracted-text hashes, although between 4.08 and 6.92 percent of pairs with nonempty sampled text fell below 0.80 Jaccard similarity. Googlebot combined the broadest access among crawler labels with the highest divergence on several representation measures, whereas OAI-SearchBot had the lowest divergence in the conditional samples. Shorter text windows and a common successful-access sample preserved this contrast. The findings describe differentiated acquisition at prominent home pages, with substantial access differences and representation differences concentrated in a minority of successful pairs.