STR-Saliency: Decomposition-based Perturbations to Generate Saliency Maps for Temporal Black-box Model Interpretation

The interpretability of deep black-box temporal models is crucial in modern machine learning. Identifying crucial time steps and temporal patterns is an important way in understanding how a black-box model makes a decision on a time series instance. Saliency methods are widely used for the interpretation of deep models. However, its application on temporal models faces two challenges: handling temporal relations and the selection of perturbation functions. In order to overcome these challenges, we propose Seasonal-Trend-Remainder Saliency (STR-Saliency), a new interpretation framework which decomposes a time series into three components then generates saliency maps on each component using a learning-based perturbation process. Our new method not only addresses the two issues above, but also produces more human-understandable saliency maps than previous methods. We evaluate our method on both synthetic and real-world datasets and it outperforms various baselines. Our code and detailed results are available at https://github.com/chenrunkai/STRSaliency.