A hybrid variational autoencoder approach to outlier detection in high-dimensional retrospective monitoring

Modern industrial and service processes generate high-dimensional, non-Gaussian, and contamination-prone data that challenge the foundational assumptions of classical Statistical Process Control (SPC). Heavy tails, multimodality, nonlinear dependencies, and sparse special-cause observations can distort baseline estimation, mask true anomalies, and prevent reliable identification of an in-control (IC) reference set. To address these challenges, we introduce VAE Self-Correcting Outlier Uncovering Technique (VSCOUT), a distribution-free framework designed specifically for retrospective (Phase I) monitoring in high-dimensional settings. VSCOUT integrates an Automatic Relevance Determination Variational Autoencoder (ARD-VAE) with ensemble-based latent outlier filtering, changepoint screening, and second-stage refinement, forming a retrospective Phase I workflow that targets pointwise and structural contamination while constructing stable high-dimensional IC baselines from historical data for future Phase II monitoring. The ARD prior isolates the most informative latent dimensions, while the ensemble and changepoint filters identify pointwise and structural contamination within the determined latent space. The second-stage refinement step removes flagged observations and re-estimates the latent structure using only the retained inliers, mitigating masking and stabilizing the IC latent manifold. This two-stage refinement produces a clean and reliable IC baseline suitable for subsequent Phase II deployment. Extensive experiments across benchmark datasets demonstrate that VSCOUT achieves superior sensitivity to special-cause structure while maintaining controlled false alarms, outperforming classical SPC procedures, robust estimators, and modern machine-learning baselines. Its scalability, distributional flexibility, and resilience to complex contamination patterns position VSCOUT as a practical and effective method for retrospective modeling and anomaly detection in AI-enabled environments.