Audio and Visual Speech for the Biometric Security System

ABSTRACT An intelligent system has been developed to enhance lip‐reading accuracy, thereby utilizing it for the biometric application, which is sometimes prone to errors due to human unpredictability. This system, known as an audio and visual biometric system, utilizes a classification‐focused supervised learning algorithm as well as estimation to predict spoken English digits based on lip movements in videos. The system uses a proprietary dataset and extracts various audio features such as Mel Frequency Cepstral Coefficients (MFCC) feature, mel feature, contrast feature, and chroma feature. For the video feature model, the system considers the differences in the location of lip points between the current and previous frames using the shape predictor. Both audiovisual dimensions have been taken from the visual dataset. Characteristics derived from both the audio and visual aspects are used for training the SVM system. The accuracy achieved by the audio‐based model is 93.03%, and the visual‐based model achieves 89.23% accuracy.