DRIVE-T: A Methodology for Discriminative and Representative Items Selection for Design Quality Constructs and Assessments

The lack of measurement constructs for both users’ literacy and artifacts difficulty in HCI hinders the quality of assessment tests. This paper proposes DRIVE-T (Discriminating and Representative Items for Validating Expressive Tests), a methodology designed to construct and evaluate items for measuring progressive levels of users’skills and artifacts usability. Given an artifact (a text, a data visualization, an interface prototype), DRIVE-T supports the identification of items discriminability and representativeness for measuring distinct traits of a quality property to be measured and its levels of progression in users and artifacts. DRIVE-T consists of three steps: (1) tagging task-based items associated with an artifact; (2) rating them by independent raters for their difficulty; (3) analyzing raters’ raw scores through a Many-Facet Rasch Measurement model. The emergence of difficulty levels of the measurement construct can be derived from the discriminability and representativeness of items for each artifact, ordered into Many-Facets construct levels. The DRIVE-T methodology operationalizes an inductive, practice-based measurement construct design. Based on the expertise of the authors in the data visualization domain, visualization literacy is the quality property of users that is exploited as a case scenario for applying DRIVE-T. Results from a pilot test show the validity of the approach.