Unpacking the grammar of tests and measures : the hidden complexity in psychological test items and its association with item responses

It has been suggested that the use of complex syntax in psychological test items may hinder some respondents’ comprehension of items (Graesser et al., 2000; Tourangeau et al., 2000). However, the prevalence and patterns of syntactic complexity features in psychological tests are not well understood, nor is their relationship with item responses. This two-part study addresses this underexplored area. Study 1 uses the Register Functional approach, which involves a corpus-based analysis, to identify the prevalence of one type of syntactic complexity, dependent clauses, in non-commercial psychological tests. It reveals nonfinite complement clauses—commonly used to express cognitive states, desires, stances, or probabilities (Biber et al., 2021)—as the most prevalent type. Study 2 then investigates the relationship between dependent clause types and item responses and subsequently examines any moderating effect of respondent’s vocabulary familiarity for associations found. Of the five dependent clause types analyzed, only nonfinite complement clauses showed a significant association with item responses, and this association was moderated by respondents’ vocabulary familiarity scores. Overall, these studies highlight the value of an interdisciplinary approach to measurement issues. The research emphasizes the importance of considering linguistic complexity in psychological test development, offering important insights into patterns of syntactic complexity in non-commercial psychological tests and the relationship between clausal complexity and response processes. By establishing foundational findings, this research sets the stage for future studies to explore the interplay between linguistic complexity and response processes, with the potential to inform test design and enhance validity and fairness for diverse populations.