Comparative Evaluation of Confidence Calibration and Selective Abstention Across Heterogeneous Security Decision Tasks

Security decision systems increasingly combine machine-learning scores with domain-specific integrity evidence, yet raw confidence values are often used without checking whether they remain reliable under operating variation. This paper presents a controlled comparative study of confidence calibration and selective abstention across three representative security settings: biometric identity verification, communicationsecurity monitoring, and digital-content integrity checking. The study does not introduce a new classifier or security architecture. Instead, it evaluates standard logistic regression, random forest, extremely randomized trees, and multilayer perceptron models on reproducible synthetic benchmarks designed to express the types of trust signals commonly reported in these domains. An isotonic calibration stage is additionally applied to the Extra Trees model, and a simple confidence-based abstention policy is evaluated under both development and shifted operating conditions. Across three random seeds and the three domains, logistic regression obtained the highest mean development macro-F1 of 0.915 and AUROC of 0.971, while calibrated Extra Trees achieved macroF1 of 0.913 and AUROC of 0.969. Calibration reduced the Extra Trees expected calibration error from 0.114 to 0.015 without materially changing discrimination. Under controlled distribution shift, calibrated Extra Trees retained macro-F1 of 0.907 and AUROC of 0.967. With a confidence threshold of 0.90, selective prediction retained about 71.2% of development decisions while increasing accuracy on retained cases to 97.4%. The results support treating calibration and abstention as separate operational properties from classification accuracy when security decisions are made from heterogeneous trust signals.

Authors

Publication Details

Journal
Research Square
Published
2026-11-20
DOI
https://doi.org/10.21203/rs.3.rs-11802873/v1
Primary Topic
Adversarial Robustness in Machine Learning
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Comparative Evaluation of Confidence Calibration and Selective Abstention Across Heterogeneous Security Decision Tasks

Raji fredo
1 links
Research Square
Adversarial Robustness in Machine Learning
preprint

Comparative Evaluation of Confidence Calibration and Selective Abstention Across Heterogeneous Security Decision Tasks

Raji fredo
preprint en

Abstract

Security decision systems increasingly combine machine-learning scores with domain-specific integrity evidence, yet raw confidence values are often used without checking whether they remain reliable under operating variation. This paper presents a controlled comparative study of confidence calibration and selective abstention across three representative security settings: biometric identity verification, communicationsecurity monitoring, and digital-content integrity checking. The study does not introduce a new classifier or security architecture. Instead, it evaluates standard logistic regression, random forest, extremely randomized trees, and multilayer perceptron models on reproducible synthetic benchmarks designed to express the types of trust signals commonly reported in these domains. An isotonic calibration stage is additionally applied to the Extra Trees model, and a simple confidence-based abstention policy is evaluated under both development and shifted operating conditions. Across three random seeds and the three domains, logistic regression obtained the highest mean development macro-F1 of 0.915 and AUROC of 0.971, while calibrated Extra Trees achieved macroF1 of 0.913 and AUROC of 0.969. Calibration reduced the Extra Trees expected calibration error from 0.114 to 0.015 without materially changing discrimination. Under controlled distribution shift, calibrated Extra Trees retained macro-F1 of 0.907 and AUROC of 0.967. With a confidence threshold of 0.90, selective prediction retained about 71.2% of development decisions while increasing accuracy on retained cases to 97.4%. The results support treating calibration and abstention as separate operational properties from classification accuracy when security decisions are made from heterogeneous trust signals.

Research Square
Adversarial Robustness in Machine Learning
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon