A Hybrid Transformer GRU Architecture for Stock Price Prediction: Optimization and Performance Evaluation Using the Artificial Rabbits Algorithm

Accurate stock price forecasting has been a well-known challenge due to strong nonlinearity, long-range temporal dependencies and high sensibility to hyper-parameters in financial time-series data. In response to these challenges, this study presents a hybrid Transformer–layered Gated Recurrent Unit (GRU) framework which has been optimized using the Artificial Rabbits Optimization (ARO) algorithm to predict next-day stock price. Transformer attention captures long-range dependencies, while GRU models temporal dynamics. The ARO algorithm further improves the performance of the model by auto-tuning training hyper-parameters such as window size, batch size, learning rate and number of epochs. The framework is evaluated on Nifty50 stock data for the period 2019–2025 and bench-marked against Artificial Neural Networks, multi-layer GRU models and GRU-GA which is a Genetic Algorithm-optimized GRU. Findings from the experiments indicated a steady and statistically significant improvement from the model, with $$R^2$$ values between 0.94 and 0.98 over various market segments and significant reductions in the MAE, MSE and MAPE. The findings confirm the efficacy of combining attention mechanisms and metaheuristic optimization and support the applicability of the proposed Transformer–layered GRU–ARO model for financial time-series forecasting in the real world. To the best of our knowledge, this is among the first studies to integrate Transformer attention with layered GRU under ARO-based hyperparameter optimization for large-scale Indian market data.