Scale Is Not All You Need: Revisiting the Biomimetic Roots of Deep Learning to Overcome Fundamental Limitations

Deep Learning has historically drawn direct inspiration from nature, with Artificial Neural Networks initially grounded in neurobiological principles. However, the field has increasingly diverged from its biomimetic origins in favor of scaling-driven approaches. While this shift has delivered remarkable technical progress, it introduced fundamental limitations relative to natural intelligence. This review distills a vast body of literature into a uniquely comprehensive synthesis of these limitations, challenging the prevailing belief that scaling current methods is sufficient for achieving human-level intelligence. Drawing on recent advances in neuroscience and deep learning, we then connect each of the limitations with promising neuro-inspired solutions. Specifically, we explore the emerging topics of Liquid Neural Networks, recurrent vision architectures, and Predictive Coding as representative examples of such solutions. A number of interesting insights emerge from the literature covered throughout this synthesis, including but not limited to the common roots of model fragility and poor generalization, relationships between model nonlinearity and adversarial robustness, parallels between Predictive Coding and Joint-Embedding Predictive Architectures (JEPA), and how backpropagation constrains more radical neuro-inspired innovations. By drawing novel connections across the covered limitations and solutions, this review bridges previously disconnected lines of research and illuminates promising next steps for neuro-inspired AI, explicitly contextualizing each direction in terms of the limitations it aims to overcome. More broadly, we hope this work offers a coherent roadmap for pushing the frontiers of artificial intelligence by reverse engineering natural intelligence.