Fraud Detection in Online Interview Using Audio Visual Synchronization

The increasing reliance on remote communication platforms for processes like job interviews necessitates robust methods to verify participant authenticity. This project proposes an automated system to detect potentially fraudulent or spoofed interviews by analyzing audio-visual synchronization cues. Manual verification methods lack scalability and struggle against advanced manipulations like deepfakes or pre-recorded responses. To address this, the system segments interview recordings into manageable clips and extracts text independently from lip movements (via lip-sync-to-text models) and spoken audio (using speech recognition). These parallel text streams are aligned using timestamped subtitles, then transformed into semantic vector embeddings. The similarity between these embeddings is measured using metrics like Cosine Similarity. A low similarity score indicates a potential manipulation. This system offers a scalable, objective, and datadriven solution for authenticating remote interviews and enhancing recruitment integrity. Keywords—Audio-Visual Synchronization, Lip Reading, Speech Recognition, Semantic Embeddings,