Estimating Correlations Across Tasks In Experimental Psychology

Understanding how people covary in performance across experimental tasks is central toindividual-difference psychology. The classic Pearson correlation has two strengths: (1) it isinvariant to the scale of measurement, and (2) it is invariant to including additionalvariables in the analysis. However, it is susceptible to attenuation from measurement noise.Bayesian hierarchical models address this issue by modeling measurement error directly.Resulting estimates, however, depend on prior specifications and are not invariant to scaleor variable inclusion. We compare three common priors—Inverse Wishart (IW), ScaledInverse Wishart (SIW), and LKJ—to assess robustness to prior assumptions in hierarchicalsettings. Our main tools are visualizing the priors and evaluating their effects on posteriorestimates through simulation. When prior settings match ground truth, all priors recovertrue correlations accurately in low-dimensional settings. When prior variance ismisspecified, the IW shows strong bias: low-variance priors inflate correlations, andhigh-variance priors deflate them. The SIW shows the same pattern but less severely, whilethe LKJ remains largely unaffected by scale misspecification. When more variables areadded, the IW is most stable, whereas the SIW and LKJ show slight shrinkage towardlower correlations. The main drawback of the LKJ is computational speed—models with itcan take orders of magnitude longer than those using IW or SIW. Overall, the LKJprovides the most accurate estimates, while the SIW offers a practical compromise forlarge-scale models where computational speed is crucial.