Establishing the reliability of metrics extracted from long-form recordings using LENA and the ACLEW pipeline

Long-form audio recordings are increasingly used to study individual variation, group differences, and many other topics in theoretical and applied fields of developmental science, particularly for the description of children's language input (typically speech from adults) and children's language output (ranging from babble to sentences). The proprietary LENA software has been available for over a decade, and with it, users have come to rely on derived metrics like adult word count (AWC) and child vocalization counts (CVC), which have also more recently been derived using an open-source alternative, the ACLEW pipeline. Yet, there is relatively little work assessing the reliability of long-form metrics in terms of the stability of individual differences across time. Filling this gap, we analyzed eight spoken-language datasets: four from North American English-learning infants, and one each from British English-, French-, American English-/Spanish-, and Quechua-/Spanish-learning infants. The audio data were analyzed using two types of processing software: LENA and the ACLEW open-source pipeline. When all corpora were included, we found relatively low to moderate reliability (across multiple recordings, intraclass correlation coefficient attributed to the child identity [Child ICC], was < 50% for most metrics). There were few differences between the two pipelines. Exploratory analyses suggested some differences as a function of child age and corpora. These findings suggest that, while reliability is likely sufficient for various group-level analyses, caution is needed when using either LENA or ACLEW tools to study individual variation. We also encourage improvement of extant tools, specifically targeting accurate measurement of individual variation.

Authors

Institutions

Publication Details

Journal
Behavior Research Methods
Published
2024-09-20
DOI
https://doi.org/10.3758/s13428-024-02493-2
Citations
6
Primary Topic
Language Development and Disorders
Type
article
Field-Weighted Citation Impact
4.74

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Establishing the reliability of metrics extracted from long-form recordings using LENA and the ACLEW pipeline

William N. Havard, Okko Räsänen, Najla Al Futaisi, Caroline F. Rowland et al.
6 citations
Behavior Research Methods
Language Development and Disorders
4.74
article

Establishing the reliability of metrics extracted from long-form recordings using LENA and the ACLEW pipeline

William N. Havard, Okko Räsänen, Najla Al Futaisi, Caroline F. Rowland, Zixing Zhang, Elika Bergelson, Alejandrina Cristià, Björn W. Schuller, Caitlin M. Fausey, Margaret Cychosz, Marvin Lavechin, Lucas Gautheron, Mélanie Söderström, Camila Scaff, Loann Peurey, Heather Anderson
article en
6 citations

Abstract

Long-form audio recordings are increasingly used to study individual variation, group differences, and many other topics in theoretical and applied fields of developmental science, particularly for the description of children's language input (typically speech from adults) and children's language output (ranging from babble to sentences). The proprietary LENA software has been available for over a decade, and with it, users have come to rely on derived metrics like adult word count (AWC) and child vocalization counts (CVC), which have also more recently been derived using an open-source alternative, the ACLEW pipeline. Yet, there is relatively little work assessing the reliability of long-form metrics in terms of the stability of individual differences across time. Filling this gap, we analyzed eight spoken-language datasets: four from North American English-learning infants, and one each from British English-, French-, American English-/Spanish-, and Quechua-/Spanish-learning infants. The audio data were analyzed using two types of processing software: LENA and the ACLEW open-source pipeline. When all corpora were included, we found relatively low to moderate reliability (across multiple recordings, intraclass correlation coefficient attributed to the child identity [Child ICC], was < 50% for most metrics). There were few differences between the two pipelines. Exploratory analyses suggested some differences as a function of child age and corpora. These findings suggest that, while reliability is likely sufficient for various group-level analyses, caution is needed when using either LENA or ACLEW tools to study individual variation. We also encourage improvement of extant tools, specifically targeting accurate measurement of individual variation.

Behavior Research MethodsVol. 56(8)
Université d'Orléans (FR), Centre National de la Recherche Scientifique (FR), Hunan University (CN), Tampere University (FI), University of Wuppertal (DE), University of Oregon (US), University of Zurich (CH), Université Paris Sciences et Lettres (FR), Harvard University Press (US), École Normale Supérieure - PSL (FR), Max Planck Institute for Psycholinguistics (NL), Laboratoire de Sciences Cognitives et Psycholinguistique (FR), University of Manitoba (CA), Imperial College London (GB), Technical University of Munich (DE), University of Maryland, College Park (US), École des hautes études en sciences sociales (FR)
National Science Foundation, James S. McDonnell Foundation, European Commission, Agence Nationale de la Recherche, Academy of Finland, China Scholarship Council, Grand Équipement National De Calcul Intensif, National Institutes of Health, Directorate for Engineering, Horizon 2020 Framework Programme, Social Sciences and Humanities Research Council of Canada, Economic and Social Research Council
Openalex Percentile: Top 5%
Language Development and Disorders
4.74
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.