Deep Learning–Based Automatic Modulation Classification Using Hybrid CNN–XGBoost Model for Wireless Communication Systems

ABSTRACT Accurate modulation classification is a key challenge in wireless communication systems, directly influencing signal decoding and system reliability. This paper presents a hybrid model that combines both convolutional neural networks (CNNs) and extreme gradient boosting (XGBoost) to enhance modulation classification performance. Firstly, CNN extracts high‐level features from input data, leveraging a fully connected architecture with dropout regularization to prevent overfitting. These features are then used by the XGBoost classifier for robust decision‐making. The proposed framework was evaluated on a modulation characteristic dataset, achieving a test accuracy of 98.3%. Performance metrics, including precision, recall, and F1 score, were calculated for each modulation class, with the average F1 score exceeding 0.9834. Furthermore, the hybrid model demonstrated resilience in noisy conditions, as shown by receiver operating characteristic (ROC) curves with the area under a curve (AUC) values greater than 0.98 for most classes. These results highlight the efficiency of the CNN–XGBoost hybrid approach in addressing complex signal classification tasks and its potential for deployment in real‐world communication systems.