Prediction of Imbalance Prices Through Gradient Boosting Algorithms: An Application to the Greek Balancing Market

An accurate prediction of imbalance prices is crucial for making well-informed decisions within short-term energy markets. This study proposes a two-stage probabilistic framework for the prediction of system imbalance and imbalance price. In the first stage, the quantiles of system imbalance are predicted employing the Quantile Regression Forest algorithm. In the second stage, the quantiles are leveraged to predict the imbalance prices, given the observed correlation between the system imbalance and the imbalance prices. The corresponding imbalance prices for each 15-minute interval are then predicted utilizing a novel methodology that combines Gradient Boosting with Markov switching models. The proposed methodology was evaluated using real data from the Greek balancing market. Comparative analysis with alternative models including a persistence model, Quantile Linear Regression, Long Short-Term Memory, and Quantile Regression Forest reveals that the proposed approach consistently outperforms the rest in terms of forecast accuracy. Specifically, the results demonstrate that the proposed approach outperforms all alternative methods, achieving an 9% improvement in nMAE and a 10.8% reduction in CRPS with respect to the second best method (Quantile Regression Forest). The findings suggest that this approach can significantly improve the decision-making process for market participants, helping mitigate the financial risks associated with imbalance prices.