Improving Automatic IPSyn Coding

Abstract

Purpose The Computerized Language ANalysis–Index of Productive Syntax (CLAN-IPSyn) system is designed to facilitate automatic computation of the IPSyn measure of productive child syntax. Roberts et al. (2020) conducted a thorough comparison of hand-generated and automatic scores on the Index of Productive Syntax (IPSyn) measure (Scarborough, 1990) and found a high level of error for CLAN-IPSyn. We report on the use of the Roberts et al. analysis to reduce and eliminate errors in CLAN-IPSyn, to improve its accuracy. Method Scores provided by manual and machine scoring of the 20 transcripts used in Roberts et al. (2020) were compared. Divergences in point assignment were examined and significant modifications made to the CLAN-IPSyn program to increase its accuracy. Conclusion The currently available, free version of CLAN at https://talkbank.org is now significantly more correct in terms of exemplars produced, and should assist clinicians and researchers in using the revised IPSyn (Altenberg et al. 2018).

Tracking Child Language Development With Neural Network Language Models

“…Despite some success in the application of automatic syntactic analysis to this task (Sagae et al, 2005 ; Hassanali et al, 2014 ; Lubetich and Sagae, 2014 ), these past efforts served more to demonstrate feasibility than to provide practical tools that can be used routinely in a variety of research situations. Roberts et al's ( 2020 ) recent effort to perform an independent evaluation of an implementation of automatic IPSyn scoring, and the subsequent effort to improve automatic scoring based on that evaluation (MacWhinney et al, 2020 ) highlight the amount of care and engineering effort required to make reliable automatic scoring widely available. The very small number of languages for which a detailed metric such as IPSyn is available further stresses the scale of the larger task of making resources available for language development research in various languages, allowing for both greater depth of language-specific findings and cross-lingual research.…”

“…Unlike in previous work to automate measurement of syntactic development (Sagae et al, 2005 ; Hassanali et al, 2014 ; MacWhinney et al, 2020 ) or to obtain a data-driven approximation to an existing metric (Lubetich and Sagae, 2014 ), the target for the scores in our model is not simply another value that can be derived for each transcript, such as an IPSyn score or age in months. Since the goal of our model is to track development over time and assign scores that reflect the chronological order of language samples for a child, we evaluate our model and compare it to baselines based on this task directly.…”

“…This work provided initial evidence that automatic IPSyn scoring was possible, and served as the basis for subsequent work to make the concept practical, for example through CLAN-IPSyn (accessible at http://talkbank.org ). These efforts have highlighted both the promise of more widespread and consistent assessment of syntactic development and the difficulty in matching the quality of analyses produced by experts (MacWhinney et al, 2020 ; Roberts et al, 2020 ).…”

The Index of Productive Syntax: Psychometric Properties and Suggested Modifications

“…Roberts et al, 2020). However, working together, MacWhinney et al (2020) were able to improve the itemlevel accuracy of CLAN's IPSyn analysis, performed on simple, uncoded transcripts of child utterances, to over 95%.…”

Dynamic Norming and Open Science

“…The comparison can be further filtered for sample size in terms of utterances, gender (male, female), activity type (narrative, interview, free play), design (cross-sectional, longitudinal), clinical status (typically developing, atypical), and comparison with an alternative age group. Because CHILDES files include automatically computed morphosyntactic analyses, the program can also automatically compute all the mean length of utterance measures used by Systematic Analysis of Language Transcripts (SALT), along with the Developmental Sentence Score (DSS; Lee, 1974 ), the Index of Productive Syntax (IPSyn; Scarborough, 1990 ; see also MacWhinney et al, 2020 ), values on the 14 grammatical morphemes studies by Brown (1973) , and several measures of lexical diversity. In all, KIDEVAL produces outcomes on 41 variables that are output to a .csv file for possible further analysis by Excel and statistical programs.…”