Mixed non linear programming model for optimizing and training long short-term memory architectures: application in traffic flow forecasting

Long short-term memory (LSTM) networks, a class of recurrent neural networks (RNNs), are widely used for sequential data modeling and time series forecasting. Their predictive accuracy, however, strongly depends on architectural choices and hyperparameter settings. In this study, we formulate LSTM architecture optimization as a mixed-variable combinatorial problem and propose a hybrid framework combining Particle Swarm Optimization (PSO) with the Adam optimizer to adaptively tune various LSTM variants, including standard LSTM, stacked LSTM (sLSTM), multiplicative LSTM (mLSTM), and extended LSTM (xLSTM). The approach is evaluated on two traffic flow datasets: the Metro Interstate dataset with daily forecasting intervals, and the PeMS datasets with one-hour ahead predictions across multiple sensor networks. Experimental results reveal a consistent performance hierarchy: the PSO-optimized xLSTM achieves the highest predictive accuracy (MAE = 257.83, RMSE = 385.97, SMAPE = 11.88%, $$R^2 = 0.961$$ on Metro Interstate; MAE = 6.056, $$R^2 = 0.992$$ on PeMS08), followed by the mLSTM, while standard LSTM and sLSTM lag behind. These findings demonstrate that framing LSTM architecture optimization as a mixed-variable combinatorial problem, coupled with PSO-based optimization, substantially improves forecasting performance, offering a robust and versatile strategy for accurate traffic prediction and other complex sequential data applications.