Automations Black-Box Conundrum: The Interpretability Crisis of Data-Driven System Identification and the Path Forward

Data-Driven System Identification (DDSI) has shone brightly in the contemporary field of automation control. Modern automated control systems now rely on DDSI as their main source of intelligence and performance enhancement. However, the "black-box" models represented by deep learning have brought about a serious interpretability crisis. Their opaque decision-making processes have severely affected the predictability, verifiability, and safety of automated control systems, which has hindered their application and trustworthiness in critical tasks of automated systems. This paper deeply analyzes the interpretability crisis faced by DDSI in automated control, especially its impact on the understanding of complex control decisions and the prediction of system dynamic behaviors. On this basis, this paper critically examines the limitations of current mainstream interpretability methods in addressing the unique dynamic challenges of automated systems, and proposes a solution path and research outlook that combines enhanced physical understanding with Explainable AI (XAL). This paper aims to provide key analysis for building more transparent, trustworthy, and safe next-generation data-driven automated control systems.