Forecasting language proficiency levels with CNN-LSTMs and real-time speech data

Accurate prediction of language proficiency levels through automatic analysis of speech signals plays a vital role in the assessment of language skills in language teaching and the development of human–machine interaction systems. Accordingly, in the present study, a new method is proposed to solve this problem by combining deep learning and ensemble learning techniques. The proposed method consists of four steps. In the first step, speech signals are preprocessed. In this process, first, the Least Mean Squares (LMS) adaptive filter is used to remove noise, and then the signal normalization and segmentation operations are performed using maximum amplitude and fixed length techniques, respectively. The preprocessed signals are processed by a model for feature representation using temporal spectral modeling and entropy features to describe salient features related to language proficiency levels. Then, two deep learning models are used to extract features from the signal and Spectro-Temporal Modulation (STM) features. The Long-Short-Term Memory (LSTM) model applies to the speech signal windows directly, and the Convolutional Neural Network (CNN) model applies to the STM feature set to obtain language skills-related features. The above two sets of features are concatenated together with entropy features, and subsequently, language levels are predicted through an ensemble classification model based on a mixture of Multilayer Perceptron (MLP) and Error-Correcting Output Codes (ECOC). The experimental results of the proposed method showed that the proposed method was able to achieve an accuracy of 89.56% and F-Measure of 0.895. This achievement indicates the high efficiency and accuracy of the model in predicting language proficiency levels through automatic analysis of speech signals.