Advanced real-time monitoring techniques for high-dimensional data streams in industrial two-sample analysis

Abstract High-dimensional data, characterized by a greater number of variables than observations, is increasingly relevant in industrial applications due to advancements in computational power and data storage. Developing control charts for such data poses challenges in statistical process control, particularly for two-sample cases where traditional feature reduction methods are insufficient. Therefore, two-sample means tests such as Srivastava and Du (SD), Dempster (DR), and Bai and Saranadasa (BS) tests effectively address high-dimensional challenges, such as the curse of dimensionality and unreliable covariance matrix estimation. The SD test modifies Hotelling’s $${T}^{2}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:msup> <mml:mrow> <mml:mi>T</mml:mi> </mml:mrow> <mml:mn>2</mml:mn> </mml:msup> </mml:math> test by assuming a diagonal covariance structure, the BS test replaces the covariance matrix with a scaled identity matrix, and the DR test employs a pseudo-inverse covariance matrix to address singularity issues. These tests are scalable, robust, and theoretically sound, outperforming traditional methods. While Shewhart control charts based on these tests detect large shifts in location parameters, they are less effective for small shifts. To overcome this, exponentially weighted moving average (EWMA) charts named SDEWMA, DREWMA, and BSEWMA were developed. Simulations and real-world high-dimensional data, such as wind turbine bearing grease damage, demonstrate the improved performance of proposed charts in detecting small shifts compared to traditional memoryless charts.