Large Language Models as Benchmarks in Forecasting Practice

Benchmark forecasting methods are important in helping us determine investments in new forecasting models. Large language models (LLMs) can now generate forecasts based on prompts, without the need for an in-depth understanding of forecasting theory, practice, or programming languages. This raises the question: Should forecasts from LLMs be a new benchmark in forecasting practice? To answer this, Hossein Hassani and Emmanuel Sirimal Silva conducted a comparative analysis. They forecasted three datasets (death series, air passengers, and UK tourist arrivals) using LLMs (ChatGPT and Microsoft Copilot) and the forecast package in R. While finding some evidence of the potential of LLMs as a forecasting tool, the authors also note challenges and limitations in their use in forecasting practice.