Self-Updating LightGBM Clustering: A Hybrid Approach for Managing Data Intermittency, Noise, and Missing Values

Despite efforts to improve the quality of time series data during collection, real-world datasets continue to face challenges such as intermittency, missing values, and noise. These issues degrade data quality and hinder the performance of artificial intelligence models, a problem pervasive across various fields including robotics, retail, finance, healthcare, and manufacturing. This paper proposes a robust solution that combines univariate fuzzy time series methods with multivariate models and AI techniques to address these common challenges in industrial datasets. Fuzzy logic is employed to effectively mitigate noise and handle intermittency in individual time series, while the multivariate model boosts performance by incorporating interdependent data to complement fuzzy logic. Implemented on a real-world dataset, our approach significantly enhances forecasting accuracy, showcasing lower root mean square error (RMSE) and mean absolute percentage error (MAPE) compared to conventional methods such as generative adversarial network (GAN), k-nearest neighbors (KNN) for outlier detection and imputation, and the use of the bidirectional long short-term memory (Bi-LSTM) model. By integrating fuzzy logic with light gradient boosting machine (LightGBM), we provide a scalable, reliable solution that tackles key data quality issues, enabling precise forecasting in noisy and intermittent environments across multiple industries.