Do Not Use Inferential Statistics to Evaluate Quality of Matching

Aims and Objectives: Quantitative research in bilingualism often involves matching between groups on one or more potentially confounding variables. A common approach to assessing the quality of matching is to apply inferential statistics to the variable(s). I argue and demonstrate that this approach is theoretically and practically problematic. Approach: I use simulated data from fictitious studies to demonstrate the problems that can arise from the use of inferential statistics for evaluating matching quality. I also analyse statistical practices for group matching in published articles from two bilingualism journals. Data and Analysis: The simulated data are analysed by t -tests, but this is merely for ease of illustration, as any inferential statistical analysis leads to the same problem. The statistical practices in published articles are analysed with logistic regression. Conclusions: There is no purely quantitative, objective alternative to inferential statistics. Matching quality can only be evaluated by a comparison of the sample distributions, informed by expert knowledge about the research topic. Originality: Although I am not the first to identify the issue, it has not yet been laid out for an audience of researchers studying bilingualism. Also, earlier attempts to improve the evaluation of matching quality proposed objective quantities as an alternative, which I argue is not viable. Significance/Implications: If researchers no longer take the outcome of an inferential statistical test as a measure of matching quality and instead present rich, descriptive statistics and embrace the subjective nature of the issue of matching, this will contribute to improving the reliability of published findings.