Calibration Quality and Query-Level Risk Detection in BM25 Retrieval

Internet-facing search and retrieval-augmented generation systems often use retrieval scores as confidence signals to route, filter, or withhold results, although ranking quality and probabilistic confidence are different properties. We test whether better-calibrated BM25 scores are more useful for such query-level decisions, using MS MARCO passages with TREC Deep Learning 2019 and 2020 judgments. Naive min–max normalization, Platt scaling, and isotonic regression are compared in a two-fold cross-collection design, and detection of below-median retrieval is evaluated with a fold-stratified area under the receiver operating characteristic curve (AUROC). Naive normalization was not calibrated across collections, and calibrating raw scores did not outperform a constant predictor at the training-set prevalence (pooled Brier score 0.211 for both). Normalizing before calibration gave the lowest Brier scores (0.201–0.202) and the lowest miscalibration in point estimate. The estimators with lower point-estimate calibration error did not demonstrate better risk detection: they reached AUROCs of 0.562–0.568, close to normalization alone (0.562) and below a top-100 score-dispersion baseline (0.731). This held for top-100 calibrated aggregates, absolute effectiveness thresholds, and calibrators refitted in the bootstrap, although moderate gains or losses cannot be excluded. Calibration quality and decision utility should be validated separately before retrieval scores serve as control signals.

Authors

Institutions

Publication Details

Journal
Future Internet
Published
2026-11-19
DOI
https://doi.org/10.3390/fi18110639
Primary Topic
Information Retrieval and Search Behavior
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Calibration Quality and Query-Level Risk Detection in BM25 Retrieval

Khaled Albishre
Future Internet
Information Retrieval and Search Behavior
article

Calibration Quality and Query-Level Risk Detection in BM25 Retrieval

Khaled Albishre
article en

Abstract

Internet-facing search and retrieval-augmented generation systems often use retrieval scores as confidence signals to route, filter, or withhold results, although ranking quality and probabilistic confidence are different properties. We test whether better-calibrated BM25 scores are more useful for such query-level decisions, using MS MARCO passages with TREC Deep Learning 2019 and 2020 judgments. Naive min–max normalization, Platt scaling, and isotonic regression are compared in a two-fold cross-collection design, and detection of below-median retrieval is evaluated with a fold-stratified area under the receiver operating characteristic curve (AUROC). Naive normalization was not calibrated across collections, and calibrating raw scores did not outperform a constant predictor at the training-set prevalence (pooled Brier score 0.211 for both). Normalizing before calibration gave the lowest Brier scores (0.201–0.202) and the lowest miscalibration in point estimate. The estimators with lower point-estimate calibration error did not demonstrate better risk detection: they reached AUROCs of 0.562–0.568, close to normalization alone (0.562) and below a top-100 score-dispersion baseline (0.731). This held for top-100 calibrated aggregates, absolute effectiveness thresholds, and calibrators refitted in the bootstrap, although moderate gains or losses cannot be excluded. Calibration quality and decision utility should be validated separately before retrieval scores serve as control signals.

Future InternetVol. 18(11)
Umm al-Qura University (SA)
Information Retrieval and Search Behavior
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon