Correlation and agreement between Language ENvironment Analysis (lena™) and manual transcription for Dutch natural language recordings

Abstract

The Language ENvironment Analysis system (LENA™) automatically analyzes the natural sound environments of children. Among other things, it estimates the amounts of adult words (AWC), child vocalizations (CV), conversational turns (CT), and electronic media (TV) that a child is exposed to. To assess LENA's reliability, we compared it to manual transcription. Specifically, we calculated the correlation and agreement between the LENA estimates and manual counts for 48 five-min audio samples. These samples were selected from eight day-long recordings of six Dutch-speaking children (ages 2-5). The correlations were strong for AWC, r = . 87, and CV, r = . 77, and comparatively low for CT, r = . 52, and TV, r = . 50. However, the agreement analysis revealed a constant bias in AWC counts, and proportional biases for CV and CT (i.e., the bias varied with the values for CV and CT). Agreement for detecting electronic media was poor. Moreover, the limits of agreement were wide for all four metrics. That is, the differences between LENA and the manual transcriptions for individual audio samples varied widely around the mean difference. This variation could indicate that LENA was affected by differences between the samples that did not equally affect the human transcribers. The disagreements and biases cast doubt on the comparability of LENA measurements across families and time, which is crucial for using LENA in research. Our sample is too small to conclude within which limits LENA's measurements are comparable, but it seems advisable to be cautious of factors that could systematically bias LENA's performance and thereby create confounds.

Validation of the Language ENvironment Analysis (LENA) system for Dutch

“…The only other languages, besides American English, for which validation of CVC has been conducted is European French and Dutch (see Table 1). Canault and colleagues (2016) (Busch et al, 2018). When combining validation studies in American English with those in other languages, a high overall correlation for CVC (mean r = .77) is reported with negative relative error rates indicating that the LENA system shows a tendency to under-estimate CVC (Cristia, Bulgarelli, et al, 2020).…”

“…related to using an automated approach in mapping descriptive measures of the home language environment of young children. The validation of the LENA System has been performed for Spanish (as spoken in the United States; Weisleder & Fernald, 2013), European French (Canault, Le Normand, Foudil, Loundon, & Thai-Van, 2016), Swedish (Schwarz et al, 2017), Chinese (Gilkerson et al, 2015), Korean (Pae et al, 2016), Vietnamese (Ganek & Eriks-Brophy, 2018b) and Dutch (Busch, Sangen, Vanpoucke, & van Wieringen, 2018). Validation for Chinese (with a focus on the Shanghai dialect and Mandarin) has shown that the LENA system can also be used in a tone language with a prosody that differs from American English, indicating that acoustic differences do not seem to invalidate system performance (Gilkerson et al, 2015).…”

“…This because specific counts seem to be influenced to a higher extent by the language that is being spoken (e.g., due to differences in speed of delivery, pitch, accent, dialectal variations, etc.) than speaker and sound categories (Busch et al, 2018;Canault et al, 2016). Defining speaker and sound categories relies mainly on acoustic features that are extracted from the recording, but can be influenced by environmental factors such as a noisy environment (Xu et al, 2008).…”

Diarization in Maximally Ecological Recordings: Data from Tsimane Children

“…Nonetheless, researchers did try to assess accuracy for their language. However, evaluation for talker diarization performance specifically also tends to be limited, as researchers' interest typically focus on global language input metrics, such as rough estimations of the number of adult words or syllables recorded in a day [11,12,13,14,15].…”

A step-by-step guide to collecting and analyzing long-format speech environment (LFSE) recordings

“…LENA's recording device also comes with software that automatically detects and classifies stretches of speech into noise-type categories (e.g., target child speech, female adult speech, etc., plus several nonspeech categories; see Ganek & Eriks-Brophy 2018a, for an overview of the system). LENA-based research has been conducted to different extents in a range of communities in North America, Europe, and East Asia; (Bergelson & Aslin, 2017;van Alphen, Meester, & Dirks, 2017;Busch, Sangen, Vanpoucke, & Wieringen et al, 2017;Schwarz, Botros, Lord, & Marcusson, 2017;Canault, Normand, Foudil, Loundon, & Thai-Van, 2016;Ganek & Eriks-Brophy 2018a;2018b;Elo, 2016;Gilkerson et al, 2015;Soderstrom & Wittebolle, 2013;VanDam, Ambrose, & Moeller, 2012;Weisleder & Fernald, 2013).…”