Development and Preliminary Evaluation of a Brief Health Chatbot Acceptability Scale (CAS): A Pilot Study

Background. Conversational agents (chatbots) are increasingly used to deliver health information to middle-aged and older adults, a population in which the determinants of technology acceptance are known to diverge from those of younger users. However, acceptability is inconsistently defined and typically measured with ad hoc, unvalidated instruments. The psychometrically sound chatbot measures that exist assess usability rather than acceptability or were validated in student samples. Objective. To develop and conduct an initial psychometric evaluation of a brief, 10-item health chatbot acceptance measure grounded in the Technology Acceptance Model (TAM) and the Theory of Planned Behavior (TPB). Methods. Twenty adults (mean age = 52.8 years, range 28-81; 85% aged 40 or older; 75% female) enrolled in a sleep-application pilot usability study completed the 10 candidate items, the System Usability Scale (SUS), and sleep measures (PROMIS Sleep Disturbance 8a; Pittsburgh Sleep Quality Index). We examined item performance, internal consistency (Cronbach's alpha), dimensionality (exploratory factor analysis with oblique rotation), and convergent validity (correlations with the SUS). Results. A two-factor structure emerged and was invariant across rotation methods, resulting in an Information Quality factor and an Acceptance factor (interfactor r = .39). All items loaded ≥ .60 on their primary factor; mean communality was .83. Subscale reliabilities were high (Information Quality α = .96; Acceptance α = .94; full scale α = .93). Both subscales correlated with the SUS in the expected direction (r = .69 and r = .59), providing preliminary support for convergent validity. Conclusions. The measure shows a coherent and theory-consistent two-factor structure and shows preliminary evidence of reliability and convergent validity. Given the small pilot sample, these results must be viewed as provisional but identify the measure as a candidate for formal validation in a larger, independent sample.