Investigation on different strategies of significance testing in ANOVA-simultaneous component analysis (ASCA)

ANOVA Simultaneous Component Analysis (ASCA) integrates analysis of variance with multivariate modelling to quantify how experimental factors and their interactions affect complex multivariate measurements. Statistical significance in ASCA is typically assessed by permutation testing; however, different permutation strategies imply distinct null hypothesis and exchangeability assumptions. In this study, we systematically compare three widely used approaches embedded in popular chemometric software packages where the permutation strategy is often predefined and not always transparent to the user. The restricted permutation method shuffles observations only within experimental strata, preserving the structure of the null hypothesis. The reduced‐model permutation contrasts the full ASCA model with a simplified version in which selected effects are removed. Permutation of marginal design matrices isolates interaction effects by permuting marginal matrices derived from the design matrix. We evaluate these methods on simulated datasets with varying patterns of main effects and interactions, as well as on an experimental study of feral cabbage (Brassica oleracea) under treatment and time factors. Our results show that the restricted permutation method reliably detects main effects, reduced‐model permutation excels at identifying interactions, and permutation of marginal design matrices consistently captures both. By examining the assumptions and performance of each method, we provide practical guidance for selecting the optimal permutation strategy in ASCA-based chemometric analysis, particularly for balanced experimental designs. As a baseline, we additionally assessed unrestricted permutation of the raw data using two test statistics: the sum of squares and the F-ratio. The results demonstrated that when employing the F-ratio, this approach was also capable of accurately detecting statistical significance. • Three ASCA permutation strategies were systematically compared • Restricted permutation best detected main effects with low Type I error • Reduced-model permutation effectively identified interaction effects • Marginal design permutation showed balanced accuracy for all effects • Practical guidance is provided for selecting ASCA permutation methods