Fractions Skill Score across spatial scales: statistical significance and discrimination

Abstract. The Fractions Skill Score (FSS) is usually interpreted through its magnitude, by comparison with reference values such as the useful-skill level (1 + f0)/2 or the expected score of a random forecast. Controlled synthetic experiments with summer convective cells, in which the correspondence between forecast and observed cells is known by construction, are used here to separate the magnitude of the score, its statistical significance against forecasts made of independently placed cells, and its ability to discriminate corresponding from unrelated forecasts, for single field pairs and for seasons of 90 aggregated pairs. Three results follow. First, reference values are not significance tests: used as decision rules for one forecast, they are exceeded by 0–99 % of unrelated forecasts depending on width, threshold, domain and number of cells, whereas Monte Carlo critical values keep false positives near 5 %. For aggregated scores, the critical value converges at rate n−1/2 to the asymptotic limit of the score under the null, and the random-forecast reference of Antonio and Aitchison is this limit for their Bernoulli null when the forecast probability equals the mean observed fraction: a central value, not a percentile. Second, the mean FSS of corresponding forecasts rises steadily with the width, whereas discrimination is largest at a width set by the position error, 5–15 grid points of 2.2 km for cells displaced by up to their radius, and declines beyond it; restricting the windows to the interior of the domain roughly doubles that decline. Third, seasonal aggregation multiplies the separation by about √n, and the most discriminating width of the aggregated score stays at or next to that of a single day: it ranks forecasting systems of different capability, but does not increase the evidence carried by the score of any single forecast. Critical values are specific to the population verified and to the null against which the score is judged; for seasons the delta method gives an asymptotic approximation to them in closed form from the daily sums of the score, without simulation. The seasonal score evaluates a forecasting system; the single-case analysis shows at which scale the score of each of its forecasts carries most evidence.

Authors

Publication Details

Published
2026-11-12
DOI
https://doi.org/10.5194/egusphere-2026-6860-v1
Primary Topic
Meteorological Phenomena and Simulations
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Fractions Skill Score across spatial scales: statistical significance and discrimination

Rodica Tomozeiu, F. Nerozzi
Meteorological Phenomena and Simulations
preprint

Fractions Skill Score across spatial scales: statistical significance and discrimination

Rodica Tomozeiu, F. Nerozzi
preprint en

Abstract

Abstract. The Fractions Skill Score (FSS) is usually interpreted through its magnitude, by comparison with reference values such as the useful-skill level (1 + f0)/2 or the expected score of a random forecast. Controlled synthetic experiments with summer convective cells, in which the correspondence between forecast and observed cells is known by construction, are used here to separate the magnitude of the score, its statistical significance against forecasts made of independently placed cells, and its ability to discriminate corresponding from unrelated forecasts, for single field pairs and for seasons of 90 aggregated pairs. Three results follow. First, reference values are not significance tests: used as decision rules for one forecast, they are exceeded by 0–99 % of unrelated forecasts depending on width, threshold, domain and number of cells, whereas Monte Carlo critical values keep false positives near 5 %. For aggregated scores, the critical value converges at rate n−1/2 to the asymptotic limit of the score under the null, and the random-forecast reference of Antonio and Aitchison is this limit for their Bernoulli null when the forecast probability equals the mean observed fraction: a central value, not a percentile. Second, the mean FSS of corresponding forecasts rises steadily with the width, whereas discrimination is largest at a width set by the position error, 5–15 grid points of 2.2 km for cells displaced by up to their radius, and declines beyond it; restricting the windows to the interior of the domain roughly doubles that decline. Third, seasonal aggregation multiplies the separation by about √n, and the most discriminating width of the aggregated score stays at or next to that of a single day: it ranks forecasting systems of different capability, but does not increase the evidence carried by the score of any single forecast. Critical values are specific to the population verified and to the null against which the score is judged; for seasons the delta method gives an asymptotic approximation to them in closed form from the daily sums of the score, without simulation. The seasonal score evaluates a forecasting system; the single-case analysis shows at which scale the score of each of its forecasts carries most evidence.

Meteorological Phenomena and Simulations
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon