Evaluating LLM-Derived Event Representations for Quantitative Trading

This study develops an LLM-augmented event-driven forecasting framework that transforms financial news into structured events, aggregates them into ticker day representations, learns event features, incorporates market regime information, and evaluates the resulting signals in long-short backtests. Using 406,286 modeled stock news records from 2009–2023, we construct 197 event features and 314 regime-aware features and conduct five annual expanding window evaluations for 2019–2023. Under a common LightGBM learner and trading protocol, structured event features deliver a substantial improvement in risk-adjusted performance relative to restricted sentiment-style inputs, with RankIC improving from −0.0128 to 0.0025 and net Sharpe from −2.267 to −0.309. Additional event principal components and regime-conditioning components provide mixed incremental benefits. Supplementary controlled comparisons do not establish a statistically reliable event-semantic gain beyond non-event firm and sector covariates. The evaluation incorporates corrected price series, delayed execution, trading frictions, multiple-comparison control, and dependent data uncertainty. A blinded human validation of 306 articles shows high semantic agreement with Qwen-plus annotations. Overall, the findings support structured LLM event representations as a promising foundation for financial forecasting while highlighting the importance of robust implementation and cost-aware evaluation.