Navigating a multimodal ensemble: Learners mediating verbal and non-verbal turns in online interaction tasks

Abstract

Research into the multimodal aspects of language is increasingly important as communication through a screen plays a greater role in modern society than ever before (Liou, 2011). Multimodality has been explored from a number of angles relating to computer-mediated communication (CMC), such as its affordances and impact on language learners, highlighting its relevance and importance in the field of second language acquisition (SLA). Because CMC scenarios require attending to both peers and the screen, learners can be seen as positioned as “semiotic initiators and responders” (Coffin & Donohue, 2014). Increasingly, researchers are highlighting a need for a methodological “turn” to analyse this scenario from a “language” focus to a more holistic understanding of the interactions (Flewitt, 2008; Hampel & Hauck, 2006; Kress & van Leeuwen, 2001; Lamy, 2006). Along these lines, this case study explores how the action of task completion is mediated between six dyads (and individuals within the dyads) during an online peer-to-peer audioconferencing event. Drawing on notions from multimodal (inter)actional analysis (Norris, 2004, 2006) and the notion of “semiotic initiators and responders”, it investigates semiotic mediation with screen-based resources through analysis of audio recordings, screenshots, log files, task simulation and reconstruction. Results highlight oral and screen-based initiations and responses that take place during task completion, which is presented as a framework.

Language learning and technology: A conceptual analysis of the role assigned to technology

“…Three of the studies in mixed-method studies were case studies (Abe, 2021;Y. C. Hsieh, 2019;Knight et al, 2020), and two of the studies were called "exploratory studies" (Andujar & Salaberri-Ramiro, 2021;Lai & Chang, 2020). One study was set up as a piece of action research (Saricaoglu & Bilki, 2021).…”

“…Participants included 66 undergraduate EFL students, 56 men and 10 females between the ages of 18 and 20 enrolled in general English classes at a Taiwanese vocational institution (Fang et al, 2021). Then it comes to the studies done with adults (Chen, 2020;Knight et al, 2020) and preschool (Alemi & Haeri, 2020;Wong & Samudra, 2021) and middle school Tan et al, 2020). At least it was done with high school (Selcuk et al, 2021) and elementary school (Shadiev et al, 2020) groups (Figure 4).…”

“…In one study (Alemi & Haeri, 2020), a robot with text-to-speech feature was used. In some studies, mobile applications such as WhatsApp (Andujar, 2020;Andujar & Salaberri-Ramiro, 2021) and KakaoTalk (Wrigglesworth, 2020), audioconferencing (Knight et al, 2020) were preferred for communication. The game named "Playing History" (Chen & Hsu, 2020) and "Pokémon Go" (Wu, 2021) were used to serve objectives of the studies.…”

Rethinking ‘Affordance’, ‘Agency’ and ‘User’ from a Semiotic Technologies Perspective: The Emergence of a Typology of Signs-as-Agents

“…Following on from the theoretical framework, Section 3 illustrates how signs (or screen-based resources) were identified and agentalised through use, which led to the key concept presented in this section (Section 2) of 'signs-as-agents'. Section 3 revisits the data from an exploratory study, spanning over 5 years, that focused initially on learner agency in an online language learning scenario to develop speaking skills [23,25,33] and evolved into a focus on 'hidden' digital agents from a more critical perspective [18].…”

“…Jovanovic and Van Leeuwen [19] also mapped out the ways that digitally-mediated interactions occur across various forms of social media, including "user-user interactions" and "system-user interactions", in what the authors describe as "a multimodal realisation of an exchange structure". This notion has also been reflected in educational research as 'semiotic initiations or responses' [24] that can be understood as part of 'multimodal turn-taking' [25]. An example of this is a pop-up that 'requests' a user to accept cookies and then the user clicks the screen-based icon as the 'acceptance'.…”