The Analytic Fluency Scale: Measuring the Skill of a Data Analyst

Psychologists should analyze their data well rather than badly, but how do we know whether they are actually good at analyzing data in the real world? Currently, there is a dearth of empirically-grounded criteria for assessing real-world analytic skill, leaving us to judge analysts’ decisions on the uncertain or even arbitrary standards of routine, disciplinary norms, or what “feels right” or “seems off”. This knowledge gap is surprising given widespread concerns about low replication, high rates of questionable research practices, and the recent proliferation of data science initiatives both within and beyond psychology purporting to increase job-readiness for data-intensive workplaces. We address this gap by introducing and validating a new construct: analytic fluency, or real-world analytic decision-making that consistently produces trustworthy, informative, and reliable analyses across contexts. Drawing on interviews with experienced analysts, we identify and validate four distinct dimensions of analytic skill: Analyzing Flexibly, Appreciating the Importance of Data, Prioritizing Stakeholder Needs, and Striving for Simplicity. We show analytic fluency scores are predicted by real analytic experience, are empirically distinct from statistical knowledge, and show no bias by gender, race/ethnicity, or education level. By naming and measuring analytic fluency, we address a longstanding conceptual and empirical gap in knowledge about how data analysis is, and ought to be, carried out in practice. In so doing, we take a first small step toward scientifically defining what it means to be skilled at data analysis.