Using fuzzy string matching for automated assessment of listener transcripts in speech intelligibility studies

Abstract

Many studies of speech perception assess the intelligibility of spoken sentence stimuli by means of transcription tasks (‘type out what you hear’). The intelligibility of a given stimulus is then often expressed in terms of percentage of words correctly reported from the target sentence. Yet scoring the participants’ raw responses for words correctly identified from the target sentence is a time-consuming task, and hence resource-intensive. Moreover, there is no consensus among speech scientists about what specific protocol to use for the human scoring, limiting the reliability of human scores. The present paper evaluates various forms of fuzzy string matching between participants’ responses and target sentences, as automated metrics of listener transcript accuracy. We demonstrate that one particular metric, the token sort ratio, is a consistent, highly efficient, and accurate metric for automated assessment of listener transcripts, as evidenced by high correlations with human-generated scores (best correlation: r = 0.940) and a strong relationship to acoustic markers of speech intelligibility. Thus, fuzzy string matching provides a practical tool for assessment of listener transcript accuracy in large-scale speech intelligibility studies. See https://tokensortratio.netlify.app for an online implementation.

Web-based language production experiments: Semantic interference assessment is robust for spoken and typed response modalities

“…However, manually classifying the correctness of typed word entries still takes a considerable amount of time (see also Borrie et al, 2019). Automated assessment of typed responses can be a highly efficient and replicable method (within and between raters) to further reduce the effort (e.g., Bosker, 2021). To test the applicability of automated assessment in typed picture naming, we compared our semi-automatic/manual classification to an automated classification procedure using the Jaro distance.…”

“…6.3;van der Loo, 2014). The metric is bounded between 0 and 1 (0 representing identical strings and 1 representing complete dissimilarity) and tailored specifically to human-typed, rather short strings (Bosker, 2021;van der Loo, 2014). 2 For the exact formula applied, we may refer to van der Loo (2014).…”

“…Different procedures exist that are specialized for different typewritten answers, such as longer texts and or single words. We recommend Bosker (Bosker, 2021) und van der Loo (2014) as overview articles. We found that the Jaro-distance (Jaro, 1989(Jaro, , 1995, a method specialized for short answers, lead to good results with a cut-off criterion of d = .3.…”

“…Before running an experiment with typewritten responses, researchers must decide whether participants typed answers should be displayed on the screen and whether participants are allowed to correct their typewritten answers by using the backspace key. Displaying participants answers on the screen can both affect naming latencies (Perret & Laganaro, 2013;Snyder et al, 2015) and error types (Pinet & Nozari, 2020, 2021. However, in online settings, we assume that giving no feedback at all may reduce adherence to the task.…”

“…They have been implemented in a wide variety of online experiments, including psycholinguistic experiments using categorization tasks (Mathôt & March, 2021;Vogt et al, 2021). Additionally, it has already been shown that typewritten texts can be preprocessed automatically and a wide range of different procedures exists, even controlling for typing errors (Borrie et al, 2019;Bosker, 2021;Navarro, 2001). Experiments with typewritten answers can thus be an easy-to-implement, time-efficient alternative to spoken responses in reactiontime sensitive language production experiments.…”