Is automatic speech-to-text transcription ready for use in psychological experiments?

Abstract

Verbal responses are a convenient and naturalistic way for participants to provide data in psychological experiments (Salzinger, 1959). However, audio recordings of verbal responses typically require additional processing such as transcribing the recordings into text, as compared with other behavioral response modalities (e.g. typed responses, button presses, etc.). Further, the transcription process is often tedious and time-intensive, requiring human listeners to manually examine each moment of recorded speech. Here we evaluate the performance of a state-of-the-art speech recognition algorithm (Halpern et al., 2016) in transcribing audio data into text during a list-learning experiment. We compare the computer-generated transcripts to transcripts made by human annotators. Both sets of transcripts matched to a high degree and exhibited similar statistical properties, in terms of the participants' recall performance and recall dynamics that the transcripts captured. This proof-of-concept study suggests that speech-to-text engines could provide a cheap, reliable, and rapid means of automatically transcribing speech data in psychological experiments. Further, our findings open the door for verbal response experiments that scale to thousands of participants (e.g. administered online), as well as a new generation of experiments that decode speech on-the-fly and adapt experimental parameters based on participants' prior responses.

Vocabulary Size Influences Spontaneous Speech in Native Language Users: Validating the Use of Automatic Speech Recognition in Individual Differences Research

“…Instead, we intend to provide a proof-of-concept that an ASR can be used to analyze certain aspects of spontaneous speech, allowing for large-scale use of natural speech for research ends. A similar approach has recently been taken by Ziman et al (2018), who showed that an ASR can be used reliably to transcribe speech data from psychological experiments, in their case a verbal recall memory test. In their study, Ziman and colleagues provided the speech context to their speech-to-text engine.…”

“…The strength of the correlations might be considered an index of how well a given measure, in the context of spontaneous speech elicitation, is suited to be transcribed by an ASR, or whether it may require manual coding. (For a similar correlational approach to evaluate transcription accuracy, see Ziman et al, 2018. ) As we planned to carry out correlations for many measures of interest, we applied a Bonferroni correction (four measures and three questions resulted in a corrected alpha level of 0.05/12 = 0.004).…”

Fitness tracking reveals task-specific associations between memory, mental health, and physical activity

“…The word lists participants studied were drawn from the categorized lists reported by Ziman et al (2018). Each participant was assigned four unique randomly chosen lists (in a randomized order), selected from a full set of 16 lists.…”

Integrating the Digital and the Traditional to Deliver Therapy for Depression

“…This could be addressed by telephone contact, but one unexpected outcome of using instant messaging was the active use by some patients of session transcripts as personalised psychoeducation materials. This benefit would be lost, although the latest advances in voice recognition and automatic transcription may eventually render this issue obsolete [20,25,81]. Access to transcripts may be particularly helpful for patients who do not engage with formal worksheets and could be actively encouraged by therapists in such situations.…”