Explainable transformer-based audio loudness estimation using acoustic feature tokens and perceptron-based regression

Over the past few years there has been growing interest in properly measuring audio loudness to provide consistent sound to a user no matter what platform they are accessing through including music streaming, broadcast, and hearing devices. Existing traditional approaches based on signal-based processing, although powerful in a controlled set-up, lack the capacity to generalize to other audio genres and to real-world practice. The study suggests a data-based framework of audio loudness estimation based on a combination of classical machine learning models and advanced deep learning networks, such as TD-transformer models (transformer-based feature-token regression model) acoustic relationships and interactions between the acoustic descriptors based on multi-head self-attention, facilitating learning of higher-order, nonlinear interactions between the features and achieving robust integrated loudness prediction. Employing the GTZAN music genre dataset containing tabular features (MFCCs, RMS energy and spectral centroid) extracted in tabular format we have trained and tested a few regression models. The obtained results demonstrate that the proposed TD-Transformer model outperformed by recording the lowest value of the MAE 0.012, RMSE 0.017, and the highest value of the $${R}^{2}$$ score 0.920, proving to be a high-capacity model in accurately predicting audio loudness and also in comparison to the best baseline model MLP which recorded an MAE of 0.0150, RMSE of 0.019, and $${R}^{2}$$ score of 0.89. To implement the model interpretability and facilitate the transparency of the decision-making, we applied an explainable AI approach of local interpretable model-agnostic explanations and Shapley additive explanations to visualize and determine how the single features in the model input affected the model prediction. The proposed framework shows best performance in loudness estimation on the GTZAN dataset and suggests the potential of transformer-based feature interaction modeling in structured audio descriptors.