Native language subtitling of educational videos: A multimodal analysis with eye tracking, EEG and self‐reports

Abstract

Language assistance becomes an integral component of the teaching–learning process in non‐English– speaking multilingual societies where education takes place predominantly in local languages. Subtitles have been found to be beneficial in facilitating understanding of English language media in such situations. However, when accompanied with subtitles, information‐rich instructional materials can add to the learner's cognitive load as their attention must be split between the content and the subtitle. To evaluate the cognitive and affective impacts of subtitle language on learning, we conducted a multimodal study using eye tracking, electroencephalogram (EEG), self‐report, and pre–post test data of 51 individuals watching a 12‐minute educational video with either no subtitles, English subtitles (L2) or subtitles in their native language (L1). We discovered (a) positive learning gains for groups with subtitles, whether L1 or L2, which suggested positive impact of subtitles on learning outcome; (b) native language subtitling yielded the highest instructional efficiency and supported effective distribution of visual attention between slide content and subtitle; (c) native language subtitles elicited the lowest cognitive load, as evidenced by both EEG measurements and self‐report data; and (d) gaze data highlighted key strategies of high and low performers during interaction with subtitled media. What is already known about this topic Use of subtitles for language acquisition is an extensively researched topic. It is well established that subtitles are beneficial for language acquisition. Use of subtitles for content learning is relatively less understood with no robust finding. What this paper adds Provides a rich literature review about the impact of subtitles, namely, beneficial, detrimental or neutral, for content learning. Offers empirical results demonstrating positive impact of native language subtitles for content learning. Adds to the limited literature of multimodal investigation of learning with subtitles. Implications for practice and/or policy English language e‐learning content designers may consider adding native language subtitles for global adoption of learning products.

Using Eye-Tracking in L2 English Teaching and Learning Research: A Review for its Implementation in Costa Rica

“…For instance, employing the visual world eye-tracking paradigm can provide insights into learners' word phonological learning (Escudero at al., 2008), age of acquisition and its influence in L2 learning (Canseco-Gonzalez et al, 2010), the role of L2 proficiency in ambiguity resolution (Ellert, 2013), and processing L2 auditory stimuli (Connell et al, 2018). It has also been shown that eye-tracking with text contributes to explore L2 topics such as L2 pronoun resolution (Roberts et al, 2008), lexical access (Dussias, 2010), L2 vocabulary learning (Godfroid et al, 2013) reading comprehension with images (Tragant-Mestres & Pellicer-Sánchez, 2019), use of subtitles for L2 content learning (Negi & Mitra, 2022), and L2 learners' reading comprehension and working memory (Huang et al, 2022). However, the studies already stated have researched L2 populations from various cultural and socioeconomic backgrounds different from the Costa Rican context.…”

“…Eye-tracking is a non intrusive experimental research technique that records eye movements when performing a task and thus the natural interaction of the eyes with a stimulus (Keating, 2013;Negi & Mitra, 2022) The human eye reveals what people are paying attention to, the cognitive effort involved when processing linguistic input (Conklin et al, 2018a), and attention-related processes (Eysenck & Keane, 2015). Eye-tracking details the moment-by-moment of visual processing tapping into cognitive processing (Just & Carpenter, 1980) and language processing.…”

“…Eye-tracking with text can also be researched in combination with audio stimuli. Negi and Mitra (2022) researched the use of subtitles in L2 content learning. Learners watched a 12-minute video, while their eyes were recorded, and they were allocated in three different groups: (1) group one watched the video with L2 subtitles, (2) group two watched the video with no subtitles, and (3) group three watched the video with L1 subtitles.…”

Language Education: Conceptual and Practical Explorations (LECPE 2022)

“…Using video clips with subtitles to teach vocabulary, students learn through visualization (Sunubi & Rustam, 2020;Teng, 2022). Thus, subtitled media is believed to be more effective than conventional methods like providing written definitions on a blackboard or in a book (Canning-Wilson, 2000;Halimah et al, 2021;Negi & Mitra, 2022;Taka, 2021). In Indonesian classrooms, vocabulary instruction mostly uses conventional methods, such as providing word meanings directly, and the media/techniques are not interesting (Wahyuni & Yulaida, 2014).…”

“…Therefore, using video clips with subtitles from YouTube videos, English vocabulary learning is very helpful in implementing integrated skill-based learning. Subtitles could contextually enhance students' understanding of English expressions and assist them in acquiring new English words and idioms (Halimah et al, 2021;Kanellopoulou, 2019;King, 2002;Negi & Mitra, 2022;Reynolds et al, 2022;Taka, 2021). Likewise, Zanón (2006) asserted that the text assists most students in monitoring the words or sentences from the dialogue that might disappear.…”