Artificial Intelligence vs. Efficient Markets: A Critical Reassessment of Predictive Models in the Big Data Era

This paper critically examines artificial intelligence applications in stock market forecasting, addressing significant gaps in the existing literature that often overlook the tension between theoretical market efficiency and empirical predictability. While numerous reviews catalog methodologies, they frequently fail to rigorously evaluate model performance across different market regimes or reconcile statistical significance with economic relevance. We analyze techniques ranging from traditional statistical models to advanced deep learning architectures, finding that ensemble methods like Extra Trees, Random Forest, and XGBoost consistently outperform single classifiers, achieving directional accuracy of up to 86% in specific market conditions. Our analysis reveals that hybrid approaches integrating multiple data sources demonstrate superior performance by capturing complementary market signals, yet many models showing statistical significance fail to generate economic value after accounting for transaction costs and market impact. By addressing methodological challenges including backtest overfitting, regime changes, and implementation constraints, we provide a novel comprehensive framework for rigorous model assessment that bridges the divide between academic research and practical implementation. This review makes three key contributions: (1) a reconciliation of the Efficient Market Hypothesis with AI-driven predictability through an adaptive market framework, (2) a multi-dimensional evaluation methodology that extends beyond classification accuracy to financial performance, and (3) an identification of promising research directions in explainable AI, transfer learning, causal modeling, and privacy-preserving techniques that address current limitations.