Weighting the weeks that matter: entropic tilting for epidemic forecast evaluation

The performance of infectious disease forecasting models is commonly calculated under the implicit assumption that forecast accuracy is equally important across time. This approach may obscure or under-weight differences in model performance during the epidemic periods that matter most for public-health decisions. Here, we introduce the weighted prequential score (WPS), a forecast evaluation framework that can emphasize epidemic periods of interest. The WPS is a weighted average of scores over time, where weights are constructed through entropic tilting which up-weights priority periods while keeping the WPS as close as possible to the arithmetic average. We demonstrate the WPS on forecasts of incident hospitalizations submitted to the CDC FluSight challenge, focusing on two priority periods: weeks of rapid growth in incidence and weeks near the seasonal peak. Across season-location-horizon pairs, doubling the weight assigned to priority periods changes the top-ranked model in 10%-26% of eligible settings, while the top-ranked model remains unchanged regardless of the amount of weight assigned in 37%-68% of settings. We demonstrate that for suitable weights, the models favored by the WPS yield an average gain in accuracy during priority periods that is at least as large as the average accuracy lost during the remaining weeks. Because weights can be defined using observed outcomes, the WPS is not a proper scoring rule. Hence, we recommend using the WPS as a retrospective evaluation diagnostic. The WPS can guide public-health practitioners in selecting models that perform well during periods most relevant to their decisions, while encouraging forecasting teams to improve accuracy during these critical weeks.

Authors

Institutions

Publication Details

Journal
medRxiv
Published
2026-11-15
DOI
https://doi.org/10.64898/2026.11.12.26367949
Primary Topic
Data-Driven Disease Surveillance
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Weighting the weeks that matter: entropic tilting for epidemic forecast evaluation

Justin Lessler, Joseph Chadi Lemaitre, Joseph R. Mathews
medRxiv
Data-Driven Disease Surveillance
preprint

Weighting the weeks that matter: entropic tilting for epidemic forecast evaluation

Justin Lessler, Joseph Chadi Lemaitre, Joseph R. Mathews
preprint en

Abstract

The performance of infectious disease forecasting models is commonly calculated under the implicit assumption that forecast accuracy is equally important across time. This approach may obscure or under-weight differences in model performance during the epidemic periods that matter most for public-health decisions. Here, we introduce the weighted prequential score (WPS), a forecast evaluation framework that can emphasize epidemic periods of interest. The WPS is a weighted average of scores over time, where weights are constructed through entropic tilting which up-weights priority periods while keeping the WPS as close as possible to the arithmetic average. We demonstrate the WPS on forecasts of incident hospitalizations submitted to the CDC FluSight challenge, focusing on two priority periods: weeks of rapid growth in incidence and weeks near the seasonal peak. Across season-location-horizon pairs, doubling the weight assigned to priority periods changes the top-ranked model in 10%-26% of eligible settings, while the top-ranked model remains unchanged regardless of the amount of weight assigned in 37%-68% of settings. We demonstrate that for suitable weights, the models favored by the WPS yield an average gain in accuracy during priority periods that is at least as large as the average accuracy lost during the remaining weeks. Because weights can be defined using observed outcomes, the WPS is not a proper scoring rule. Hence, we recommend using the WPS as a retrospective evaluation diagnostic. The WPS can guide public-health practitioners in selecting models that perform well during periods most relevant to their decisions, while encouraging forecasting teams to improve accuracy during these critical weeks.

medRxiv
University of North Carolina at Chapel Hill (US)
Data-Driven Disease Surveillance
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon