The Gut Check: On Stability and Homogeneity Methods for Reference Materials

Abstract Homogeneity and stability analyses are broad topics that are essential across various scientific fields in ensuring product reliability and validity. This paper reviews traditional methods for homogeneity and stability analyses, such as Analysis of Variance (and extensions of Analysis of Variance applied to distance metrics) and trend analysis, and highlights their limitations when applied to high-dimensional, non-normal, or temporal data. Using a novel Reference Material from the National Institute of Standards and Technology, we illustrate the challenges and downfalls of applying conventional methods in characterizations of homogeneity and argue for a more intuitive and informative approach. We introduce an approach tailored for Reference Materials by examining the differences between any two samples, that can be used for either homogeneity or stability studies, without relying on hypothesis testing. Our proposed coefficient of disagreement method offers an alternative framework for assessing homogeneity and stability. This approach aims to provide more interpretable evaluations of sample variability and consistency, ultimately enhancing the robustness of data analyses in complex research contexts.