Enhancing Physical‐Layer Security in Millimeter Wave Communication Networks: Federated Learning–Assisted Key Generation and Transmission

ABSTRACT Multiple input multiple output‐orthogonal frequency division multiplexing (MIMO‐OFDM) communication, allowing high data rates and low latency, is a crucial enabler for 5G and beyond wireless networks. However, its sensitivity to eavesdropping and physical‐layer attacks poses serious security challenges. This research proposes a novel framework that enhances physical layer security in MIMO by integrating deep learning–based key generation and federated learning–assisted secure transmission. Capitalizing on the spatial uniqueness of mmWave propagation, we employ a cross‐layer mobilenetV2 pyramid mutual attention network to extract rich physical‐layer features—such as angle of arrival and angle of departure, for robust key generation without requiring prior key exchange. To enhance the security of the transmission process, we introduce a multichannel squeeze‐and‐excitation combined network trained in a federated learning setting, allowing distributed devices to jointly learn a secure transmission model without compromising data privacy. Experimental results demonstrate that the proposed method significantly enhances resistance to passive eavesdropping and ensures low bit error rates, even under dynamic channel conditions and high secrecy rate, key generation rate of about 1.63 at 40 dB signal‐to‐noise ratio and bit error rate of 0.1 at 25 dB. The proposed approach enables lightweight, scalable, and privacy‐preserving security in mmWave‐enabled communication systems.