Capacity Claims Across Specified Variation: How the Range of Conditions Tested Bounds What a Capacity Claim Can Say

This conceptual paper introduces Capacity Claims Across Specified Variation, an evaluative criterion for assessing what demonstrated success warrants saying about an agent’s capacity. A claim should specify the success criterion, the conditions tested, the reliability observed across them, and the assumptions supporting extension beyond those conditions. Assessment keeps the success criterion fixed while varying questions, collaborators, settings, or other relevant conditions. Failures remain part of the reported reliability. Conditions present throughout testing belong to the context of observed success, without their causal contribution being established by those observations alone. Generalization beyond the tested range requires stated assumptions. The discussion relates the criterion to external validity and examines Oskar Pfungst’s investigation of Clever Hans. Performance changed when questioner knowledge and the horse’s visual access were varied. These comparisons supported a narrower capacity claim; the explanation involving bodily cues depended on further investigation. Part of the Possibility and Counterfactual Structure series within On Human Modeling, the paper offers questions about what stayed constant, what varied, and who determined the permitted support. Its framework can inform human–AI research on whether capability claims remain warranted across changes in prompts, support, collaborators, and settings.