Adaptive hybrid convolutional neural network-autoencoder framework for backdoor detection in GenAI-driven semantic communication systems

Semantic communication systems, powered by Generative AI (GenAI), enable the efficient transmission of semantic meaning rather than raw data. However, these systems remain highly vulnerable to backdoor attacks, which embed malicious triggers into training datasets, causing the misclassification of poisoned samples while leaving clean inputs unaffected. Existing detection mechanisms often modify model structures, degrading clean inference performance, or impose strict data format constraints that limit adaptability. Moreover, many approaches rely on fixed similarity thresholds, making them ineffective against adaptive backdoor attacks and unable to inspect hidden activations where backdoors are embedded. To address these challenges, we propose a hybrid framework, CNN-AAE, which combines a Convolutional Neural Network (CNN) with an Adaptive Autoencoder (AAE) to leverage both spatial feature learning and semantic deviation analysis for robust backdoor detection. Unlike prior methods, our approach preserves the original model structure, dynamically adjusts detection thresholds, and analyzes internal layer activations to identify deeply embedded backdoors. We evaluate CNN-AAE on the MNIST and CIFAR-10 datasets and compare its performance against several State-Of-The-Art (SOTA) baselines, including CNN, Multilayer Perceptron (MLP), Fully Connected Neural Network (FCNN), autoencoder, and the Anti-Backdoor Model (ABM). The results demonstrate that CNN-AAE consistently achieves higher detection accuracy and significantly lower attack success rates, while maintaining efficient resource usage in terms of training time and memory consumption.