Physical-Measure Merton Jump-Diffusion Calibration Surrogates and Direct Realized Volatility Forecasting: An Empirical and Numerical Audit on S&P 500 ETF (SPY) Returns

Abstract Continuous-time Geometric Brownian Motion (GBM) fails to capture the leptokurtic tails and sudden price discontinuities observed in equity index returns. While Merton’s (1976) Jump-Diffusion (MJD) model accounts for sudden shocks by superimposing a compound Poisson jump process onto Brownian diffusion, rolling physical-measure (\(\:P\)) Maximum Likelihood Estimation (MLE) suffers from severe finite-sample weak identification, Kiefer (1978) mixture singularities, and iterative computational overhead. In this study, we design, audit, and evaluate a rigorous, leakage-free dual-task quantitative pipeline on daily S&P 500 ETF (SPY) returns (\(\:N=\text{1,237}\) validated sessions, 2019–2023). We explicitly separate two research questions that are frequently conflated in student and practitioner implementations: (Task 1) whether Ridge regression and hybrid sequence-summary Long Short-Term Memory (LSTM) networks can serve as fast calibration surrogates for rolling 126-session multistart compound-Poisson MLE, and (Task 2) whether direct supervised volatility models or calibrated total-volatility proxies can forecast forward 21-session realized volatility better than classical historical persistence and RiskMetrics EWMA (\(\:{\lambda\:}_{ewma}=0.94\)) baselines. Across 211 chronological calibration windows (\(\:134\) Fit, \(\:31\) Validation, \(\:46\) Test) separated by an explicit \(\:H=21\)-session forward-label purge, raw five-parameter MJD fits hit box boundaries in \(\:165/211\) windows (\(\:78.2\%\)), while a conservative Bayesian Information Criterion (BIC) parsimony rule against an exact nested two-parameter Gaussian submodel (\(\:\lambda\:=0\)) selects jumps in \(\:69/211\) windows (\(\:32.7\%\), concentrated entirely around the 2020 COVID-19 crash). Online surrogate inference achieves a \(\:63.9\times\:\) wall-clock speedup (\(\:0.359\:ms\) for Mean, Ridge, and LSTM combined versus \(\:22.939\:ms\) median multistart MLE), but 5-session parameter persistence (\(\:{\widehat{\theta\:}}_{t}={\widehat{\theta\:}}_{t-5}\)) outperforms ML surrogates on rolling 126-session labels (validation standardized MSE \(\:0.0268\) vs. \(\:1.8052\) for Ridge and \(\:2.2362\) for LSTM) due to \(\:96\%\) window overlap. On the held-out 2023 test split (\(\:n=46\)), the direct hybrid LSTM achieves the lowest future 21-session realized volatility RMSE (\(\:0.026553\), MAE \(\:0.023191\), Campbell–Thompson \(\:{R}_{oos}^{2}=+16.95\%\), \(\:r=0.5049\)), followed by direct Ridge (\(\:RMSE=0.028088\), \(\:{R}_{oos}^{2}=+7.07\%\)) and EWMA (\(\:RMSE=0.028179\), \(\:{R}_{oos}^{2}=+6.46\%\)), outperforming 21-session historical volatility (\(\:RMSE=0.029137\)). However, because chronological validation during the high-volatility 2022 bear market selected the 126-session calibrated total-volatility proxy (mle_total_vol, validation \(\:RMSE=0.056528\)), which subsequently lagged the 2023 volatility decline (\(\:test\:RMSE=0.050488\), \(\:{R}_{oos}^{2}=-200.26\%\)), freezing model selection prior to test evaluation prevents claiming consistent out-of-sample forecasting superiority without a fresh post-2023 holdout.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-17
DOI
https://doi.org/10.21203/rs.3.rs-11729845/v1
Primary Topic
Financial Risk and Volatility Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Physical-Measure Merton Jump-Diffusion Calibration Surrogates and Direct Realized Volatility Forecasting: An Empirical and Numerical Audit on S&P 500 ETF (SPY) Returns

Mayank Goswami, Rajdeep Purohit
1 links
Research Square
Financial Risk and Volatility Modeling
preprint

Physical-Measure Merton Jump-Diffusion Calibration Surrogates and Direct Realized Volatility Forecasting: An Empirical and Numerical Audit on S&P 500 ETF (SPY) Returns

Mayank Goswami, Rajdeep Purohit
preprint en

Abstract

Abstract Continuous-time Geometric Brownian Motion (GBM) fails to capture the leptokurtic tails and sudden price discontinuities observed in equity index returns. While Merton’s (1976) Jump-Diffusion (MJD) model accounts for sudden shocks by superimposing a compound Poisson jump process onto Brownian diffusion, rolling physical-measure (\(\:P\)) Maximum Likelihood Estimation (MLE) suffers from severe finite-sample weak identification, Kiefer (1978) mixture singularities, and iterative computational overhead. In this study, we design, audit, and evaluate a rigorous, leakage-free dual-task quantitative pipeline on daily S&P 500 ETF (SPY) returns (\(\:N=\text{1,237}\) validated sessions, 2019–2023). We explicitly separate two research questions that are frequently conflated in student and practitioner implementations: (Task 1) whether Ridge regression and hybrid sequence-summary Long Short-Term Memory (LSTM) networks can serve as fast calibration surrogates for rolling 126-session multistart compound-Poisson MLE, and (Task 2) whether direct supervised volatility models or calibrated total-volatility proxies can forecast forward 21-session realized volatility better than classical historical persistence and RiskMetrics EWMA (\(\:{\lambda\:}_{ewma}=0.94\)) baselines. Across 211 chronological calibration windows (\(\:134\) Fit, \(\:31\) Validation, \(\:46\) Test) separated by an explicit \(\:H=21\)-session forward-label purge, raw five-parameter MJD fits hit box boundaries in \(\:165/211\) windows (\(\:78.2\%\)), while a conservative Bayesian Information Criterion (BIC) parsimony rule against an exact nested two-parameter Gaussian submodel (\(\:\lambda\:=0\)) selects jumps in \(\:69/211\) windows (\(\:32.7\%\), concentrated entirely around the 2020 COVID-19 crash). Online surrogate inference achieves a \(\:63.9\times\:\) wall-clock speedup (\(\:0.359\:ms\) for Mean, Ridge, and LSTM combined versus \(\:22.939\:ms\) median multistart MLE), but 5-session parameter persistence (\(\:{\widehat{\theta\:}}_{t}={\widehat{\theta\:}}_{t-5}\)) outperforms ML surrogates on rolling 126-session labels (validation standardized MSE \(\:0.0268\) vs. \(\:1.8052\) for Ridge and \(\:2.2362\) for LSTM) due to \(\:96\%\) window overlap. On the held-out 2023 test split (\(\:n=46\)), the direct hybrid LSTM achieves the lowest future 21-session realized volatility RMSE (\(\:0.026553\), MAE \(\:0.023191\), Campbell–Thompson \(\:{R}_{oos}^{2}=+16.95\%\), \(\:r=0.5049\)), followed by direct Ridge (\(\:RMSE=0.028088\), \(\:{R}_{oos}^{2}=+7.07\%\)) and EWMA (\(\:RMSE=0.028179\), \(\:{R}_{oos}^{2}=+6.46\%\)), outperforming 21-session historical volatility (\(\:RMSE=0.029137\)). However, because chronological validation during the high-volatility 2022 bear market selected the 126-session calibrated total-volatility proxy (mle_total_vol, validation \(\:RMSE=0.056528\)), which subsequently lagged the 2023 volatility decline (\(\:test\:RMSE=0.050488\), \(\:{R}_{oos}^{2}=-200.26\%\)), freezing model selection prior to test evaluation prevents claiming consistent out-of-sample forecasting superiority without a fresh post-2023 holdout.

Research Square
Indian Institute of Technology Roorkee (IN)
Financial Risk and Volatility Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon