XGBoost-enabled low-complexity approach for signal integrity assessment in coherent communication systems

Abstract Error vector magnitude (EVM) provides critical information for assessing signal integrity and system performance in optical communication systems. In this research, a regression approach using boosting algorithms is developed to retrieve EVM information from complex signal constellation structures applicable to various modulation formats. Amplitude histograms are created and collected at different OSNR levels, launch powers, and transmission distances using an offline preprocessing approach. The impact of various evaluation techniques, including mean absolute error, coefficient of determination, root mean square error (RMSE), and mean absolute percentage error (MAPE) is discussed in detail. The results show that the proposed extreme gradient boosting framework considerably increases estimation precision compared to categorical boosting and light gradient boosting machine, especially under diverse transmission conditions, with MAPE staying below 1.7% and RMSE below 0.5, thereby enhancing overall performance monitoring in optical communication systems. This research provides a comprehensive and accurate representation of the proposed EVM estimation technique, making it a valuable resource for the development of advanced optical performance monitoring systems in the future.