Taking Language Samples Home: Feasibility, Reliability, and Validity of Child Language Samples Conducted Remotely with Video Chat Versus In-Person

Abstract

<p>Purpose: There has been a significant increased interest in using telepractice for involving more diverse children in research and clinical services, as well as when in-person assessment is challenging, such as during COVID-19. Little is known, however, about the feasibility, reliability, and validity of language samples when conducted via telepractice. Method: Child language samples from parent-child play were recorded either in person in the lab or via video chat at home, using parents’ preferred commercially available software on their own device. Samples were transcribed and analyzed using SALT software. Analyses compared measures between-subjects for 46 dyads who completed video chat language samples versus 16 who completed in-person samples; within-subjects analyses were conducted for a subset of 13 dyads who completed both types. Groups did not differ significantly on child age, sex, or socio-economic status. Results: The number of usable samples and percent of utterances with intelligible audio signal did not differ significantly for in-person versus video chat language samples. Child speech and language characteristics (including MLU, TTR, NDW, grammatical errors/omissions, and child speech intelligibility) did not differ significantly between in-person and video chat methods. This was the case for between-group analyses and within-child comparisons. Further, transcription reliability (conducted on a subset of samples) was high and did not differ between in-person and video chat methods. Conclusions: This study demonstrates that child language samples collected via video chat are largely comparable to in-person samples in terms of key speech and language measures. Best practices for maximizing data quality for using video chat language samples are provided.</p>

Exploring the validity and reliability of online assessment for conversational, narrative, and expository discourse measures in school-aged children

“…They confirmed that more rigorous statistics are needed to either confirm or dispute this preliminary evidence. More recently, Manning et al (2020) looked at the feasibility, reliability, and validity of obtaining language samples remotely by recording child-parent play with toddlers. They compared online and in-person groups on language sample metrics such as mean length of utterance (MLU), number of different words (NDW), and type-token ratio (TTR).…”

Disruption Leads to Methodological and Analytic Innovation in Developmental Sciences: Recommendations for Remote Administration and Dealing With Messy Data

“…Another important theme for developmental scientists to consider as we move past the pandemic is what research measures are "good enough" to answer the questions of interest (Blackwell et al, 2020;Morris et al, 2020). Whereas a study may have previously collected an in-depth lab-based assessment designed to measure a specific construct, the pandemic has forced researchers to reconsider whether a shortened, remote, or less burdensome method (e.g., a questionnaire) can fill that position (e.g., Manning et al, 2020). This will be an important theme moving forward, as what is most pragmatic or efficient has long been ignored in many developmental studies in favor of what is most in-depth.…”

“…One critical role that research assistants took responsibility for was providing technological support during these visits, as the platform functioning was essential to ensuring fidelity of administration, accuracy of scoring, and child compliance with the tasks. One method for addressing this issue before a problem was evident was to discuss the caregivers' comfort level with technology during the visit-scheduling phone calls (e.g., Manning et al, 2020). The research assistants provided support ahead of the visit regarding how to access the Zoom link and what kind of device/setup would be ideal to help alleviate caregiver anxiety and reduce troubleshooting in the moment of the visit.…”

Using Animated Action Scenes to Remotely Assess Sentence Diversity in Toddlers

“…A language sample at approximately 30 months of age also is included because measures of language growth help reveal when an individual toddler's pattern of language development begins to show signs of recovery from the early language delay or indicators of increased risk for DLD. To eliminate the need for families to travel to the laboratory, we collected 30-month parent–toddler language samples via remote assessment following the procedures of Manning et al (2020). We also needed a remote alternative to the examiner–child language sampling protocol that created play-based opportunities for toddlers to produce sentences with diverse subjects and verbs typically completed in the laboratory.…”

“…Language sampling is a child-friendly method for obtaining repeated measures of global language production such as the number of different words and mean length of utterance. Importantly, language measures obtained remotely via video chat are comparable with those collected in person (Manning et al, 2020). However, it can be challenging for clinicians to create sufficient opportunities for children to produce grammatical structures that occur less frequently (Hesketh, 2004;Rice et al, 1995), which is especially important for clinical assessment of grammatical structures affected by DLD.…”