Identifying storytelling in job interviews using deep learning

Structured interviews often include past-behavior questions inviting applicants to recount a past work experience. While optimal responses to these questions should take the form of a story, applicants struggle to produce them extemporaneously. Asynchronous video interviews (AVIs) present new opportunities for job interview coaching, which can incorporate artificial intelligence to analyze audio-recorded responses and deliver personalized feedback. We explore the potential of audio-based deep-learning models to identify storytelling and other, sub-optimal responses (pseudo-stories, decontextualized self-descriptions) from interview audio recordings. Using data from 254 mock interviews featuring three past-behavior questions, we developed models to determine the utterance type, considering different scenarios and labeling schemes of varying granularity. We further applied multiple techniques to improve the model accuracy. Findings show that our models achieve satisfactory performance when enhanced with audio information and enriched with longer context (best accuracy: 77.67%) However, providing paralinguistic cues from the audio recordings did not help improve the models’ performance. We discuss the results, implications, and future research directions. • A pipeline of deep learning models was developed for interview recording analysis. • Audio-based deep-learning models can identify interview response types. • Audio-based models lag behind transcript-based ones without improvement techniques. • Context is crucial for audio-based models’ performance.