Ensemble learning-based anomaly detection in coherent optical communication systems

Ensuring signal integrity in coherent systems remains critical, particularly in mitigating the effects of noise, impairments, and disruptions across advanced modulation formats. This research explores ensemble learning-based anomaly detection strategies to address challenges such as feature complexity and class imbalance. Isolation forest, one-class support vector machine, and local outlier factor are utilized as base models, whose outputs are further integrated using ensemble techniques, specifically stacking and voting, to enhance detection performance. The study rigorously evaluates the effectiveness of these ensemble methods across 16, 32, and 64 quadrature amplitude modulation using various performance metrics. The results demonstrate that integrated approaches significantly outperform individual models, achieving higher robustness, reliability, and generalization capabilities. By emphasizing the synergistic advantage of combining multiple anomaly detection algorithms, this work contributes to the advancement of real-time monitoring and anomaly management in coherent optical systems, providing important guidance for future intelligent network architecture and performance enhancement.