Enhancing return forecasting using LSTM with agent-based synthetic data

Financial markets, as complex adaptive systems, are characterized by historical data limitations, inherent evolution and non-stationarity, which challenge the effectiveness of deep learning models such as Long Short-Term Memory (LSTM). We address these challenges by generating synthetic data using Agent-Based Modeling (ABM) to simulate complex market conditions through “what-if” scenarios. Our method comprises three steps: (i) pre-training the LSTM model on historical data, (ii) generating synthetic data with the ABM using “what-if” scenarios, and (iii) fine-tuning the pre-trained LSTM with ABM-generated synthetic data. The results show that ABM-generated data significantly improve model performance across various statistical and economic metrics and are robust to diverse market environments, model architectures, and data frequencies. Our primary contribution is modeling the properties of complex adaptive systems with ABM-generated data, highlighting the need for new complex scenarios to better simulate future market conditions that are distinct from historical trends. We explore the potential of ABM in generating unique synthetic data, offering a framework to address the challenges imposed by the complex adaptive system properties of financial markets, particularly, improving the discriminative ability of forecasting models such as the LSTM model. • Our transfer learning framework integrates Agent-Based Models (ABMs) with Long Short-Term Memory (LSTM) networks to enhance financial return forecasting. • By introducing more frequent and pronounced tail events, ABM synthetic data fills a gap in tail-risk information, thereby enhancing the forecasting model's ability to handle various market conditions. • Implemented mixed variance and adjusted agents' chartist weights addresses variability and discrepancies between synthetic and historical datasets.