Divergence-Based Event Detection in Microbatch Processing for Data Streams

Bearing fault detection is critical for ensuring the reliability of rotating machinery and preventing costly breakdowns. This article presents a novel approach combining microbatch processing (MBP) with a retrospective divergence-based event detection algorithm to address key challenges in real-time fault detection, including handling multimodal sensor data, managing class imbalance, and detecting subtle fault signatures. MBP splits continuous data streams into smaller, manageable batches for near-real-time processing, but we hypothesize that this discretization can compromise detection accuracy, especially with multimodal data. To overcome these limitations, our method introduces: first, the extraction of higher order features, such as skewness and kurtosis, from microbatches to enhance the system's ability to detect early-stage faults; second, the application of Kullback–Leibler and Pearson divergence measures to detect changes in data distributions across microbatches; and third, validation using machine learning models—Naïve Bayes, decision trees, and support vector machines—to assess the algorithm's effectiveness. Experimental results demonstrate that the proposed method improves fault detection accuracy, particularly in detecting early faults and handling imbalanced data. Our findings suggest that combining MBP with retrospective divergence-based techniques is a robust solution for detecting faults in multimodal data streams, making it well-suited for real-time industrial monitoring.