Developing NOVA – A Next-Generation Open Vocabulary Assessment

Abstract: In psychological assessment, vocabulary tests are commonly used as reliable and efficient indicators of crystallized intelligence, as retrospective proxies for premorbid intelligence, and as measures of language proficiency. However, many of the widely used German vocabulary tests are outdated, proprietary, and lack a clear rationale for item selection. To address these limitations, we developed a new, openly available vocabulary test: the Next-Generation Open Vocabulary Assessment (NOVA). We constructed 110 multiple-choice vocabulary items with support from ChatGPT and administered them to 1,052 German-speaking adults using a multiple-matrix design, along with a declarative knowledge test for validation purposes. Using Ant Colony Optimization, we assembled two parallel 30-item short forms by optimizing reliability as well as item difficulty and discrimination parameters within a three-parameter logistic item response model. The resulting test forms provided unidimensional and reliable measurement, covered a broad ability range, showed no gender differences, and correlated strongly with declarative knowledge. A Shiny app is available to calculate norm-referenced scores based on individual test results. Additional analyses revealed that 61.2% of the variance in item difficulty was explained by word frequency and word length, underscoring their potential utility in guiding future vocabulary test development.