Improving Open-Set Recognition in Long-Tailed Underwater Optical Monitoring via Frequency-Shrunk Prototypes and Group-Conditional Risk Calibration

Camera-based underwater optical monitoring enables persistent and non-invasive observation of marine communities, but its automated interpretation is challenged by two coupled uncertainties: the natural long-tailed frequency distribution of observed organisms and the appearance of categories absent from model development. Rare known categories provide insufficient observations for reliably estimating class location and dispersion, whereas dominant categories may form broad acceptance regions that absorb previously unseen organisms. We propose Frequency-Shrunk Prototype and Risk Calibration (FSPRC), a known-only post hoc framework for long-tailed underwater optical monitoring built on an ImageNet-pretrained ConvNeXt-Tiny. FSPRC regularizes each empirical feature direction toward its corresponding classifier direction and shrinks its intraclass radius toward a frequency-group reference, with both corrections controlled by the number of training observations. Local prototype deviation and global energy evidence are then robustly normalized within the predicted head, middle, or tail group and combined without a trainable fusion module. Independent known-only calibration subsets determine group-conditional rejection thresholds, while held-out categories remain excluded from backbone adaptation, checkpoint selection, geometric estimation, score normalization, and threshold fitting. On the predefined fold-0 leakage-controlled protocols for AQUA20 and the track-disjoint Fish4Knowledge-23 dataset, FSPRC achieves open-set accuracies of 83.31% and 98.19%, open-set classification rate (OSCR) values of 85.96% and 98.96%, and unknown category macro area under the receiver operating characteristic curve (AUROC) values of 92.62% and 99.55%, respectively. Across exhaustive class-held-out partitions, open-set accuracy is 83.25±0.24% over four AQUA20 folds and 98.18±0.08% over five Fish4Knowledge-23 folds. These results demonstrate that frequency-dependent geometric stabilization and risk calibration can improve the recognition-stage balance between retaining data-scarce known organisms and referring unfamiliar observations for subsequent expert inspection.