Young Learners’ Processing of Multimodal Input and Its Impact on Reading Comprehension

Abstract

Theories of multimedia learning suggest that learners can form better referential connections when verbal and visual materials are presented simultaneously. Furthermore, the addition of auditory input in reading-while-listening conditions benefits performance on a variety of linguistic tasks. However, little research has been conducted on the processing of multimedia input (written text and images) with and without accompanying audio. Eye movements were recorded during young L2 learners’ (N = 30) processing of a multimedia story text in reading-only and reading-while-listening conditions to investigate looking patterns and their relationship with comprehension using a multiple-choice comprehension test. Analysis of the eye-movement data showed that the presence of audio in reading-while-listening conditions allowed learners to look at the image more often. Processing time on text was related to lower levels of comprehension, whereas processing time on images was positively related to comprehension.

Multimodal Input in Sla Research

“…It also highlights the role of tests that target different aspects of L2 knowledge development (e.g., Wisnieska & Mora, 2020) and that reveal partial learning gains (e.g., Feng & Webb, 2020;Puimège & Peters, 2020). We also find two examples of how eye-tracking can be used to study multimodal input and which types of eye-tracking metrics we might need to consider (Lee & Révész, 2020;Pellicer-Sánchez et al, 2020). While eye-tracking is a particularly interesting technique to study multimodal input because it provides an online unobtrusive account of learners' attention allocation, the multimodal character of the input also has a number of implications for data collection procedures and analyses.…”

“…First, it illustrates how different data-collection techniques ranging from paperand-pencil tests (e.g., comprehension or vocabulary tests), interviews, or questionnaires (e.g., on language exposure) to more recent techniques such as eye-tracking can be used to investigate different components of L2 learning from multimodal input. More particularly, the special issue provides interesting examples of fine-grained (Pujadas & Muñoz, 2020) and multimodal comprehension tests (Pellicer-Sánchez et al, 2020). It also highlights the role of tests that target different aspects of L2 knowledge development (e.g., Wisnieska & Mora, 2020) and that reveal partial learning gains (e.g., Feng & Webb, 2020;Puimège & Peters, 2020).…”

“…Multimodal input is not only beneficial for intermediate to advanced students (e.g., Lee & Révész, 2020;Puimège & Peters, 2020;Wisnieska & Mora, 2020) but can also be used with young and adolescent beginning L2 learners (Pellicer-Sánchez et al, 2020;Pujadas & Muñoz, 2020). Both reading while listening (with supportive images) as well as videos with subtitles in the L1 have the potential to motivate beginning L2 learners to engage with this type of L2 input and facilitates their comprehension of those materials.…”

“…Studies have, for instance, used eye-tracking to investigate whether input enhancement in the form of visually enhanced target items or salient grammatical structures draws learners' attention to target items (Lee & Révész, 2018;Montero Perez et al, 2015). Multimodal input also offers multiple options to study learners' attention allocation to different types of materials such as reading-whilelistening (Serrano & Pellicer-Sánchez, 2019) or captioned video (Montero Perez, 2019a) to investigate how learners divide their attention over different input modes (e.g., text and images) and how learners' attention allocation relates to linguistic outcomes (see Pellicer-Sánchez et al, 2020).…”

“…Studies have investigated the value of audio-visual input (Feng & Webb, 2020;Puimège & Peters, 2020), audio-visual input with subtitles in the L1 (Pujadas & Muñoz, 2020), subtitles in the L2 (Lee & Révész, 2020;Pujadas & Muñoz, 2020), and L2 subtitles with enhanced (i.e., highlighted) target constructions (Lee & Révész, 2020). It also offers a study on audio-assisted reading activities, that is, reading while listening (Pellicer-Sánchez et al, 2020). In what follows, I discuss some of the main implications 1 of the aforementioned studies for theory, methodology, and pedagogy.…”