One size fits all? In search of the desirable caption display for second language learners with different caption reliance in listening comprehension

Abstract

Existing research has established captions as effective second-language (L2) or foreign language (FL) listening comprehension aids. However, due to the transient nature of captions, not all learners are capable of attending to captions in all cases. Previous work posited that to leverage the impact of technologies in learning and instruction, a better understanding of the interplay between technology and cognition is warranted. In this vein, the current study set out to investigate the effects of four different caption modes (full vs. partial vs. real-time vs. control) on the listening comprehension of 95 high-intermediate Taiwanese learners of English as a foreign language (EFL) with different caption reliance (i.e. more-caption-reliant vs. less-caption-reliant). The results showed no significant difference between the participants’ listening comprehension outcomes under the four caption conditions when their caption reliance was not considered. However, when this was considered, the differences among the four caption conditions became salient, which was suggestive of the selective effect of captions on L2 learners with different caption reliance. While less-caption-reliant L2 learners had the best listening comprehension outcome under the partial-caption condition and the worst under the full-caption condition, more-caption-reliant L2 learners exhibited the best performance under the full-caption condition yet the worst under the partial-caption condition. The finding underscores the importance of considering L2 learners’ processing profiles when utilizing captioned videos as multimodal instructional/learning materials and speaks to the need of utilizing differentiated video materials for optimal listening outcomes.

Second or foreign language learning through watching audio-visual input and the role of on-screen text

“…Similar findings regarding the effects of keyword captions were reported by Teng (2019a) in a study with primary school learners. Additionally, Lee et al (2021) took into account learners’ self-reported caption reliance (i.e., how much they rely on captions while watching video) when comparing the effects of full, partial, and no captions on L2 learners’ comprehension. It was found that learners in a high captioning reliance group (see Section 6.2) performed best on a comprehension test after watching an 11-minute TED-talk with full captions (as compared with partial and no captions).…”

Effects of modality preference and working memory capacity on captioned videos in enhancing L2 listening outcomes

“…Specifically, post-study retrospection can allow a researcher to accurately report on one's modality preference in real-time video processing only when and if swift processing of video content is reliably open to consciousness. However, as seen in Lee et al (2019), L2 learners' modality preference while viewing the video material is not always consistent with their offline verbal report conducted after the video viewing task. Lee et al found that notwithstanding L2 learners' superior performance under the partial captioning viewing conditionan indication of their preference for partial captioning while viewing the video materialsthey found captioning distracting in the post-study retrospective questionnaire.…”

“…This study demonstrates the intimate relationship between modality reliance and multimodal L2 listening. Recently, drawing on the CRT, Lee et al (2019) examined whether learners' modality preference while viewing captioned videos affected their listening outcomes. Lee et al observed that due to different modality preference (visual vs. auditory), L2 learners showed differential reliance on captioned texts; this determined the extent to which L2 learners may have benefited from a captioning viewing environment (see also Hasler, Kersten & Sweller, 2007).…”

“…In addition to its effectiveness in promoting L2 listening outcomes, Chen, Liu and Todd (2018) and Pujadas and Muñoz (2019) both provided positive empirical evidence supporting the potential of full captions for the acquisition of vocabulary forms (see also Suárez & Gesa, 2019). Despite the above positive evidence, a few studies have suggested that the effectiveness of full captions may be modulated by L2 learners' modality preference (visual vs. auditory; see Lee, Liu & Tseng, 2019) and individual differences in preference for visual input (i.e. captions; Lee & Révész, 2018).…”

“…visual vs. audio). In this case, those who prefer auditory input may not consistently attend to captions, and those who prefer visual input may be more likely to allocate their attention to captions (Lee, Liu & Tseng, 2019). Drawing on eye-tracking technology, Lee and Révész (2018) found that L2 learners who were more caption-reliant spent more time fixating on visual input (such as captioned text) and exhibited greater learning gains after viewing captioned videos.…”