An Experimental Study on the Interpretability of Transformer Models for Dialog Understanding

Identifying the intention of an utterance (e.g., spoken sentence) is part of semantic understanding. Transformer models offer promising performance for intention identification, however model interpretability remains low. This paper presents a comprehensive experimental study on the interpretability of BERT-like models, such as DistilBERT, for spoken dialog intention understanding. A detailed discussion explains the main features used in intention classification. This insight can be a starting point to devise more interpretable transformer models.