Accuracy of Statistical Inferences Drawn from Two-Sample Graphs

What sorts of graphical formats best convey effect size and degree of certainty of a find- ing? Confidence intervals are commonly used to show uncertainty, yet lay people and experts fail to correctly interpret their meaning. There has been a recent push to present individual data points rather than only presenting aggregated summary statistics (e.g., means, confi- dence intervals, lines of best fit). But it is unclear how well people can aggregate raw data presented in a graphical format. Across two studies, we presented participants with hypo- thetical study outcomes of two independent groups in three graph styles: dot plots, mean with 95% confidence interval (CI) plots, combined plots, and bee plots. We asked partici- pants to make judgments about the effect size, using the Common Language Effect Size, or Bayes Factors. Participants were more likely to underestimate effect sizes and Bayes Factors for dot plots and bee plots compared to mean + 95% CI plots and combined plots. These findings suggest that people have trouble making statistical inferences when presented with raw data points in graphs.