Statistical Validation of Multivariate Treatment Effects in Longitudinal Study Designs

ABSTRACT Multivariate extensions of repeated measures linear mixed models, such as repeated measures ANOVA simultaneous component analysis (RM‐ASCA+) and linear mixed model‐principal component analysis (LiMM‐PCA), can be used for analyzing longitudinal studies with multivariate outcomes. However, there are no gold standards to assess the statistical validation of the observed effects of such models. Using real and simulated data, we here perform an empirical comparison of different strategies for assessing statistical significance in these frameworks: permutation tests, the global log‐likelihood ratio (GLLR) test, and nonparametric bootstrap confidence intervals for the estimated multivariate effects. Power curves were used to examine the statistical power of the different tests in detecting time–treatment interactions with varying effect sizes. Our results show that both the permutation tests and the GLLR test can be used to statistically test the presence of a time–treatment interaction effect for multivariate data; however, the GLLR approach will be sensitive to the number of included principal components in LiMM‐PCA. The bootstrap confidence interval approach generally shows good statistical power but has inflated Type 1 error rates under certain conditions. This makes it unsuitable for the purpose of hypothesis testing in its present implementation, although it may still be useful for exploratory purposes. Overall, our results show that the power of the tests for assessing multivariate effects in longitudinal studies is dependent on characteristics of the dataset, and it is important to be aware of the strengths and weaknesses of the different validation procedures.