Partial and synchronized captioning: A new tool to assist learners in developing second language listening skill

Abstract

This paper introduces a novel captioning method, partial and synchronized captioning (PSC), as a tool for developing second language (L2) listening skills. Unlike conventional full captioning, which provides the full text and allows comprehension of the material merely by reading, PSC promotes listening to the speech by presenting a selected subset of words, where each word is synched to its corresponding speech signal. In this method, word-level synchronization is realized by an automatic speech recognition (ASR) system, dedicated to the desired corpora. This feature allows the learners to become familiar with the correspondences between words and their utterances. Partialization is done by automatically selecting words or phrases likely to hinder listening comprehension. In this work we presume that the incidence of infrequent or specific words and fast delivery of speech are major barriers to listening comprehension. The word selection criteria are thus based on three factors: speech rate, word frequency and specificity. The thresholds for these features are adjusted to the proficiency level of the learners. The selected words are presented to aid listening comprehension while the remaining words are masked in order to keep learners listening to the audio. PSC was evaluated against no-captioning and full-captioning conditions using TED videos. The results indicate that PSC leads to the same level of comprehension as the full-captioning method while presenting less than 30% of the transcript. Furthermore, compared with the other methods, PSC can serve as an effective medium for decreasing dependence on captions and preparing learners to listen without any assistance.

A comprehensive word difficulty index for L2 listening

“…We generated partial and synchronized captions for the annotated TED talks using the proposed score function (Score-L), an improved version of the score using Gaussian kernel (Score-K) and expert rules (PSC1) described in Mirzaei et al (2017) for intermediate L2 learners. The number of shown words in PSCs using proposed scores are kept consistent with PSC1.…”

“…1) based on word frequency, speech rate, and specificity. Our earlier studies using these features revealed that PSC leads to the same level of comprehension as the full-caption, but better prepares learners for listening without using any textual clues (Mirzaei et al, 2017).…”

Automatic Compression of Subtitles with Neural Networks and its Effect on User Experience

“…The study was conducted in a computer lab at the university in a supervised environment using an online survey. 7 First, the participants are briefed about the study purpose and sign a form of consent. Second, they answer demographic questions (occupation, age, gender, mother tongue, English proficiency) and state their daily-life exposure to English videos and their subtitle usage behaviour with possible answers ranging from "1never" to "5 -always" (5-point Likert scale) [34].…”

“…Thus, automatic systems for compressing the spoken content are desired. Such systems could be used in various places, such as foreign language courses [6,7], lectures, talks or meetings.…”

“…Thus, additional to an evaluation on a benchmark dataset for comparison to state of the art, we assess the effect of compressed subtitles on comprehension and cognitive load in a user study. Related user studies regarding partial and keyword captions provide controversial results: Guillory [21], Rooney [22] and Mirzaei et al [7] find positive effects of partial captions and no significant differences in comprehension compared to full captions. They argue based on the dual-coding theory that as they lower the input on the visual channel, the cognitive load has to be smaller.…”