Model Evaluation for Radio-Frequency Signal Modulation Classifiers in the Existence of Novel Samples

We present an open-set recognition (OSR) pipeline for radio-frequency (RF) signal modulation that extends the recently proposed varMax uncertainty framework to the communications domain. Leveraging a multi-domain convolutional neural network (CNN) that takes in four representations of an electromagnetic signal in the form of raw In-phase and Quadrature (IQ) values and three transformations, Fast Fourier Transform (FFT), Discrete Cosine Transform (DCT), and polar magnitude, we train on eight signal modulations and treat the ninth modulation as unknown. Per fold, 1000 epochs are used to train each classifier on a unique set of eight modulations. Our approach supports the reliable rejection of unseen modulations on most folds while preserving in-distribution classification accuracy. However, the effectiveness is highly dependent on the makeup of the open and closed sets, and even seemingly inconsequential changes have a significant impact on the model's performance. This study demonstrates the strengths and weaknesses of extending the varMax approach to the RF domain and introduces modifications that improve the model's performance when trained on heterogeneous data.