Out-of-Distribution Detection Leveraging Denoising Diffusion Probabilistic Model for ISAC Systems

Integrated Sensing and Communication (ISAC) systems represent an evolution in modern wireless networks, co-designing radar and communication functions on a shared hardware and spectrum platform. While ISAC delivers higher spectral efficiency and reduced size by unifying waveforms, it introduces new challenges in non-cooperative scenarios, such as electronic warfare and spectrum surveillance, where the receiver lacks prior knowledge of signal parameters. In such contexts, out-of-distribution (OOD) detection is essential as the first line of defense to flag OOD waveforms before they reach downstream tasks, like automatic modulation classification. To address this, we propose a generative OOD detection framework for non-cooperative ISAC. Our method ingests smoothed pseudo-Wigner-Ville distribution and in-phase/quadrature constellation images of radar and communication signals, then processes them through a denoising diffusion probabilistic model (DDPM) with a U-Net backbone. DDPMs decompose data generation into denoising steps, enabling modeling of manifolds, and yield a denoising loss that is low for in-distribution but high for OOD waveforms. We interpret this loss as an OOD score and set the operating threshold via Youden’s J statistic to optimize detection trade-offs. Experimental results across diverse non-cooperative ISAC scenarios demonstrate that our DDPM-based detector outperforms conventional OOD methods, underscoring its robustness for blind estimation tasks.