The Struggles of Developing an Operational Machine Learning Model for Flare Forecasting: Recasting the Problem as a Regression onto X-Ray Flux

Abstract Complex machine learning (ML) models do not seem to have improved solar flare forecasting. Despite many efforts to apply deep neural network architectures, it is unclear whether these models truly offer improvements compared to classical statistical and ML techniques because consistent benchmark test sets are underused and few papers offer comparisons to baseline models. In particular, predicting the onset and decline of flaring activity is an unsolved challenge. One possibility for the lack of improvement in performance is that the flare forecasting problem has been cast as one of binary classification, which forces models to try to distinguish between flares below and above an arbitrary intensity threshold. Here, we define a fully regression-based forecasting task and present results for full-disk forecasts of the maximum X-ray flux within a given forecast window using models trained on historical magnetogram data. We compare various ML architectures and show that a hybrid convolutional neural network only offers comparable performance to a simple random forest model. This demonstrates the continued value of a small set of predictive scalar features in flare forecasting and the importance of comparing to a baseline. By showing how performance varies depending on the choice of test set, we also emphasize the need to simulate operational conditions when evaluating forecasting models. We do not claim to have developed a more skillful flare forecasting model in this work; rather, we seek to highlight key challenges and best practices within the context of investigating a regression task for flare prediction.