Comparing predictive ability in the presence of instability over a very short time

Summary We consider forecast comparison in the presence of instability when this affects only a short period of time. We demonstrate that global tests do not perform well in this case because they were not designed to capture very short-lived instabilities, and their power vanishes altogether when the magnitude of the shock is very large. We then discuss non-parametric approaches that are more suitable to detect such situations. We illustrate these results in a Monte Carlo exercise and in a comparison of the nowcast of the quarterly US nominal GDP from the Survey of Professional Forecasters against a naive benchmark of no growth, over a period that includes the GDP instability brought by the COVID-19 crisis. We recommend that forecasters do not pool the sample, but exclude the short periods of high local instability from the evaluation exercise.