Neural-Assisted Detector with Low False-Alarm Rate for Dense Multi-Target Sensing in OTFS-ISAC Systems

Dense multi-target sensing in orthogonal time–frequency space (OTFS)-based integrated sensing and communication (ISAC) is challenged by sub-resolution target spacing, unknown target cardinality, and inter-target radar-cross-section (RCS) variation. This paper develops a neural-assisted detector (NAD) that combines convolutional-neural-network candidate generation with continuous delay–Doppler refinement, physics-based validation, active splitting, and explicit model-order control. The CNN generates candidates from the nominal delay–Doppler grid while predicting fractional offsets, so that the final target positions are continuous-valued rather than confined to grid centers. Isolated candidates are refined individually, whereas closely spaced candidates are jointly optimized through alternating position and complex-gain updates, with active splitting used to resolve dense target pairs. For unknown target number, NAD-Hybrid combines conservative candidate generation with information-criterion-based weak-target rescue and backward pruning to control false alarms under RCS mismatch. On a frozen 8000-frame blind test covering two scenes and four RCS standard deviations, NAD-Hybrid achieves a macro-averaged F1 score of 0.9338 and reduces false alarms relative to the unknown-order OMPFR-BIC baseline, at the cost of lower detection probability under severe RCS mismatch. The proposed detector therefore provides a conservative operating point that prioritizes low false alarms and reliable target-count estimation. The sensing framework is further demonstrated to coexist with an index-modulated OTFS communication link using a shared frame.