Designing An Automated Assessment of Public Speaking Skills Using Multimodal Cues

Chen, Lei; Feng, Ganjun; Leong, Chee Wee; Joe, Jilliam N.; Kitchen, Christopher; Lee, Chong Min

doi:10.18608/jla.2016.32.13

Cited by 10 publications

(13 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Except from predicting affective states and performance and explaining engagement, other studies employed multimodal data for other research objectives, as well, including modeling dialogue Grafsgaard, Lester, & Boyer, 2015;Worsley, 2018), idea creation (Furuichi & Worsley, 2018) or motivational intentions (Yu et al, 2018), assessing presentation skills (Chen et al, 2016;Ochoa et al, 2018) and predicting collaborative coordination/synchrony between the collaborating peers (eg, Grafsgaard, Duran, Randall, Tao, & D'Mello, 2018;Schneider & Blikstein, 2015;Stewart, Keirn, & D'Mello, 2018;Worsley, 2014). Furthermore, substantial work has been done in the area of providing feedback using data in one or more modalities (Pardo, Poquet, Martínez-Maldonado, & Dawson, 2017).…”

Section: Related Work: Utilizing Multimodal Data To Predict Learning mentioning

confidence: 99%

Building pipelines for educational data using AI and multimodal analytics: A “grey‐box” approach

Sharma¹,

Papamitsiou²,

Giannakos³

2019

Brit J Educational Tech

View full text Add to dashboard Cite

Students' on‐task engagement during adaptive learning activities has a significant effect on their performance, and at the same time, how these activities influence students' behavior is reflected in their effort exertion. Capturing and explaining effortful (or effortless) behavior and aligning it with learning performance within contemporary adaptive learning environments, holds the promise to timely provide proactive and actionable feedback to students. Using sophisticated machine learning (ML) algorithms and rich learner data, facilitates inference‐making about several behavioral aspects (including effortful behavior) and about predicting learning performance, in any learning context. Researchers have been using ML methods in a “black‐box” approach, ie, as a tool where the input data is the learner data and the output is a given class from the chosen construct. This work proposes a methodological shift from the “black‐box” approach to a “grey‐box” approach that bridges the hypothesis/literature‐driven (feature extraction) “white‐box” approach with the computation/data‐driven (feature fusion) “black‐box” approach. This will allow us to utilize data features that are educationally and contextually meaningful. This paper aims to extend current methodological paradigms, and puts into practice the proposed approach in an adaptive self‐assessment case study taking advantage of new, cutting‐edge, interdisciplinary work on building pipelines for educational data, using innovative tools and techniques. What is already known about this topic Capturing and measuring learners' engagement and behavior using physiological data has been explored during the last years and exhibits great potential. Effortless behavioral patterns commonly exhibited by learners, such as “cheating,” “guessing” or “gaming the system” counterfeit the learning outcome. Multimodal data can accurately predict learning engagement, performance and processes. What this paper adds Generalizes a methodology for building machine learning pipelines for multimodal educational data, using a modularized approach, namely the “grey‐box” approach. Showcases that fusion of eye‐tracking, facial expressions and arousal data provide the best prediction of effort and performance in adaptive learning settings. Highlights the importance of fusing data from different channels to obtain the most suited combinations from the different multimodal data streams, to predict and explain effort and performance in terms of pervasiveness, mobility and ubiquity. Implications for practice and/or policy Learning analytics researchers shall be able to use an innovative methodological approach, namely the “grey‐box,” to build machine learning pipelines from multimodal data, taking advantage of artificial intelligence capabilities in any educational context. Learning design professionals shall have the opportunity to fuse specific features of the multimodal data to drive the interpretation of learning outcomes in terms of physiological learner states. The constraints from th...

show abstract

Section: Related Work: Utilizing Multimodal Data To Predict Learning mentioning

confidence: 99%

Building pipelines for educational data using AI and multimodal analytics: A “grey‐box” approach

Sharma¹,

Papamitsiou²,

Giannakos³

2019

Brit J Educational Tech

View full text Add to dashboard Cite

show abstract

“…Temporal collaborative multimodal data 1529 motivation). For example, to predict students' performance in terms of recall, quality, correctness or self-assessment, researchers have used different combinations of brain, behavioral or body signals, as well as audio cues and learning artifacts (Beardsley, Hernández-Leo, & Ramirez-Melendez, 2018;Chen et al, 2016;Di Mitri et al, 2017;Junokas, Lindgren, Kang, & Morphew, 2018;Spikol, Ruffaldi, Dabisias, & Cukurova, 2018). Combinations of different data streams yielded accurate predictions for a range of performance measures.…”

Section: Practitioner Notesmentioning

confidence: 99%

“…As can be inferred from the above examples, the data modalities used in individual and collaborative MMLA range greatly with log data, clickstreams, audio, video, dialogs, facial expressions, gestures, posture, motion, gaze and biological being most common (Beardsley et al, 2018;Chen et al, 2016;Junokas et al, 2018;Liu & Stamper, 2017;Mattingly et al, 2019;Smith et al, 2016;Spikol et al, 2018). From these data streams, a wide range of higher level features can be inferred including affect, attention, cognitive processing, stress and fatigue.…”

Section: Practitioner Notesmentioning

confidence: 99%

Temporal analysis of multimodal data to predict collaborative learning outcomes

Olsen¹,

Sharma²,

Rummel³

et al. 2020

Brit J Educational Tech

View full text Add to dashboard Cite

The analysis of multiple data streams is a long‐standing practice within educational research. Both multimodal data analysis and temporal analysis have been applied successfully, but in the area of collaborative learning, very few studies have investigated specific advantages of multiple modalities versus a single modality, especially combined with temporal analysis. In this paper, we investigate how both the use of multimodal data and moving from averages and counts to temporal aspects in a collaborative setting provides a better prediction of learning gains. To address these questions, we analyze multimodal data collected from 25 9–11‐year‐old dyads using a fractions intelligent tutoring system. Assessing the relation of dual gaze, tutor log, audio and dialog data to students' learning gains, we find that a combination of modalities, especially those at a smaller time scale, such as gaze and audio, provides a more accurate prediction of learning gains than models with a single modality. Our work contributes to the understanding of how analyzing multimodal data in temporal manner provides additional information around the collaborative learning process.

show abstract

“…Considering assessment duties, diverse issues should also be taken into account such as individual accountability, time, validity, and reliability. Concerning speaking skills, they are one of the complex assessment targets that claims novel solutions, such as the Scoring Model based on Multimodal Data (e.g., speech delivery and basic features of speech content, hand, body, and head movements) that significantly predicts human rating (Chen et al, ).…”

Section: Applicationsmentioning

confidence: 99%

Learning analytics: A glance of evolution, status, and trends according to a proposed taxonomy

Peña-Ayala

2018

WIREs Data Min & Knowl

View full text Add to dashboard Cite

Before the emergence of computer‐based educational systems (CBES) whose aims of providing teaching and learning experiences to hundreds even thousands of users, an explosion of information (e.g., students' log data) demands sophisticated methods to gather, analyze, and interpret learners' traces to regulate and enhance education. Thus, learning analytics (LA) arises as a knowledge discovery paradigm that provides valuable findings and facilitates stakeholders to understand the learning process and its implications. Therefore, a landscape of the LA nature, its underlying factors, and applications achieved is outlined in this paper according to a suggested LA Taxonomy that classifies the LA duty from a functional perspective. The aim is to provide an idea of the LA toil, its research lines, and trends to inspire the development of novel approaches for improving teaching and learning practices. Furthermore, the scope of this review covers recently published papers in prestigious journals and conferences, where the works dated from 2016 are summarized and those corresponding to 2014–2015 are cited according to the proposed LA taxonomy. A glimpse is sketched of LA, where underlying elements frame the field foundations to ground the approaches. Moreover, LA strengths, weaknesses, challenges, and risks are highlighted to advice how the LA arena could be enhanced and empowered. In addition, this review offers an insight of the recent LA labor, as well as motivates readers to enrich the LA achievements. This work promotes the LA practice giving an account of the job being achieved and reported in literature, as well as a reflection of the state‐of‐the‐art and an acumens vision to inspire future labor. This article is categorized under: Application Areas > Education and Learning Application Areas > Science and Technology Fundamental Concepts of Data and Knowledge > Human Centricity and User Interaction

show abstract

Designing An Automated Assessment of Public Speaking Skills Using Multimodal Cues

Cited by 10 publications

References 27 publications

Building pipelines for educational data using AI and multimodal analytics: A “grey‐box” approach

Building pipelines for educational data using AI and multimodal analytics: A “grey‐box” approach

Temporal analysis of multimodal data to predict collaborative learning outcomes

Learning analytics: A glance of evolution, status, and trends according to a proposed taxonomy

Contact Info

Product

Resources

About