Challenges and Methods in Annotating Natural Speech for Neurolinguistic Research

Spoken language is central to human communication, influencing cognition, learning, and social interactions. Despite its spontaneous nature, characterized by disfluencies, fillers, self-corrections and irregular syntax, it effectively serves its communicative purpose. Understanding how the brain processes natural language offers valuable insights into the neurobiology of language. Recent neuroscience advancements allow us to study neural processes in response to ongoing speech, requiring detailed, time-locked descriptions of speech material to capture the nuances of spoken language. While there are many speech-to-text tools available, obtaining a time-locked true verbatim transcript, reflecting everything that was uttered, requires additional effort to achieve an accurate representation. We demonstrate the challenges involved in the process of obtaining time-resolved annotation of spontaneous speech, by presenting two semi-automatic pipelines, developed for German and Hebrew but adaptable to other languages. The outputs of these pipelines enable analyses of the neural representation and processing of key linguistic features. We discuss the methodological challenges and opportunities posed by current state-of-the-art pipelines, and advocate for new lines of natural language processing research aimed at advancing our understanding of how the brain processes everyday language.