Dataset customization for English words recognition by articulation

Automatic speech recognition by articulation (lip reading) remains a relevant problem, despite the existing solutions. One of the key difficulties is the insufficient and imbalanced training data, which reduces the accuracy, and most importantly, the generalizing ability of the models. Purpose: to increase the generalizing ability of the automatic speech recognition system by articulation by expanding the database and fine-tuning the model. Methods: expanding the LRW dataset, containing videos of individual words, by processing and adapting the LRS2 dataset, containing video of phrases. The neural network architecture used is a combination of modified ResNet-18 and BiGRU. Results: expanding the dataset with visually similar samples from LRS2 dataset reduced the class imbalance and increased the amount of training data. Fine-tuning the model on a mixed dataset increased its accuracy and generalizing ability compared to the baseline, bringing its application closer to real-world conditions. Practical relevance: the developed method can be applied to improve automatic speech recognition systems in data-limited scenarios, namely, the customized LRS2 dataset can be used similarly and in combination with the LRW dataset. Discussion: a promising direction for future research is further dataset expansion using additional sources, as well as the development of new neural network architectures for efficient lip-reading.