Unveiling learning trends in convolutional neural networks with dynamic mode decomposition

The widespread adoption of pre-trained models in artificial intelligence poses several evaluation challenges, primarily due to limited access to comprehensive information including training data, test data, loss function, and hyperparameter values. Traditional evaluation metrics, like accuracy, precision, and recall, demand the availability of train and test data, making it difficult to determine the expected performance or quality of the deep learning models. This paper addresses this challenge by introducing a comprehensive empirical analysis of Convolutional Neural Networks (CNNs) using Dynamic Mode Decomposition (DMD) theory. Feature maps from the convolutional layers are used to model the trained CNN as a linear dynamical system. By constructing snapshots for DMD analysis, we decompose the system’s dynamics into individual modes and eigenvalues, providing a linear approximation of the intricate CNN behavior. We also introduce DMD-based metrics, including entropy of DMD-eigenvalues (EDMD), DMD eigen gap/spectral gap, and zero amplitude DMD modes, to quantify the effectiveness and quality of the learned CNN (well-trained vs poorly-trained) without the need to access the test data. Results indicate that the entropy of DMD-eigenvalues (EDMD) correlates well with the classical evaluation metrics and also distinguishes between well-trained and poorly-trained models. Additionally, we explored the potential of interpreting CNNs through stable DMD modes and their corresponding eigenvalues. Our approach was thoroughly evaluated on both shallow and deep models across various datasets for classification and segmentation tasks, demonstrating its effectiveness in quantifying CNN learning evolution and task performance. We also conducted a comprehensive analysis to examine the correlation between the proposed metrics and classical evaluation metrics on pre-trained models, specifically using ImageNet for classification (20 models) and Pascal VOC for segmentation (3 models) tasks. The code is available at: https://github.com/sikha2552/DMD_XAI . • Propose a DMD formulation for CNNs as a linear dynamical system using feature maps extracted from the filters of the convolutional blocks to construct snapshots for DMD analysis. This approach aims to analyze and interpret the behavior of CNNs and their learning evolution by decomposing the system’s dynamics into individual modes using DMD. These modes are then employed to evaluate and understand the network’s performance over epochs. • Introduce DMD-based metrics to quantify the quality of the learned CNNs, which includes: Entropy of DMD-eigenvalues (EDMD), DMD Eigen Gap/ Spectral Gap, Zero Amplitude DMD Modes, and analyze their correlation with the classical task-specific evaluation metrics (test accuracy, IoU, Dice). • Performs an extensive evaluation of the proposed quality metrics across various shallow and deep classification and segmentation models trained end-to-end on diverse datasets. Additionally, conduct a comprehensive analysis to explore the correlation between the proposed metrics and classical evaluation metrics on pretrained models, specifically using ImageNet for classification and the Pascal VOC dataset for segmentation tasks. • Present a novel approach for interpreting CNNs using the stable DMD modes and their associated DMD-eigenvalues. Compare the interpretation maps generated by the proposed DMD-based method with state-of-the-art CNN interpretability methods, including Grad-CAM, Grad-CAM++, and ScoreCAM.