Improving Affect Detection in Game-Based Learning with Multimodal Data Fusion

Henderson, Nathan; Rowe, Jonathan P.; Paquette, Luc; Baker, Ryan S.; Lester, James C.

doi:10.1007/978-3-030-52237-7_19

Cited by 27 publications

(30 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…While depictions of early and late fusion styles have been relatively consistent across multiple papers [9,17,37,41,75,84,119], there are still cases where other terms have been used. In [31], one network architecture described as multi-view-one-network is essentially early fusion and one-view-one-network could be considered late fusion.…”

Section: Applying the Taxonomymentioning

confidence: 97%

Multimodal Classification: Current Landscape, Taxonomy and Future Directions

Sleeman¹,

Kapoor²,

Ghosh³

2021

Preprint

View full text Add to dashboard Cite

Multimodal classification research has been gaining popularity in many domains that collect more data from multiple sources including satellite imagery, biometrics, and medicine. However, the lack of consistent terminology and architectural descriptions makes it difficult to compare different existing solutions. We address these challenges by proposing a new taxonomy for describing such systems based on trends found in recent publications on multimodal classification. Many of the most difficult aspects of unimodal classificationhave not yet been fully addressed for multimodal datasets including big data, class imbalance, and instance level difficulty. We also provide a discussion of these challenges and future directions.

show abstract

Section: Applying the Taxonomymentioning

confidence: 97%

Multimodal Classification: Current Landscape, Taxonomy and Future Directions

Sleeman¹,

Kapoor²,

Ghosh³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…It is also interesting to note the fusion of multimedia data (audio and video) together with data from students' hands while they performed certain tasks (Worsley, 2014). The only researchers who contributed with several papers to this survey were Henderson et al, whose studies focused on student posture and movement along with other data that varied from one study to the next (N. Henderson et al, 2020; N. L. Henderson, Rowe, Mott, & Lester, 2019). It was also interesting to see the fusion between physical and digital data gathered via webcam and the learning platform in Ma et al (2015).…”

Section: Multimodal Datamentioning

confidence: 99%

“…Five of the studies which appeared in the early fusion category stand out for going beyond simple concatenation of features with rather more detailed procedures. Four of those studies were configured to select the best features of each data source (Chango, Cerezo, & Romero, 2021; Chango, Cerezo, Sanchez‐Santillan, et al, 2021; N. L. Henderson, Rowe, Mott, Brawner, et al, 2019; N. Henderson et al, 2020). In contrast, N. L. Henderson, Rowe, Mott, and Lester (2019), reduced the dimensionality of the features using principal component analysis (PCA) in two different configurations: (a) they concatenated all of the features of the sources and applied PCA to the resulting vector; (b) they applied PCA to the features of each source first and concatenated the results following the reduction of dimensionality.…”

Section: Data Fusion Techniques In Multimodal La/edmmentioning

confidence: 99%

A review on data fusion in multimodal learning analytics and educational data mining

Chango

Lara²,

Cerezo

et al. 2022

WIREs Data Min & Knowl

View full text Add to dashboard Cite

The new educational models such as smart learning environments use of digital and context-aware devices to facilitate the learning process. In this new educational scenario, a huge quantity of multimodal students' data from a variety of different sources can be captured, fused, and analyze. It offers to researchers and educators a unique opportunity of being able to discover new knowledge to better understand the learning process and to intervene if necessary. However, it is necessary to apply correctly data fusion approaches and techniques in order to combine various sources of multimodal learning analytics (MLA). These sources or modalities in MLA include audio, video, electrodermal activity data, eye-tracking, user logs, and click-stream data, but also learning artifacts and more natural human signals such as gestures, gaze, speech, or writing. This survey introduces data fusion in learning analytics (LA) and educational data mining (EDM) and how these data fusion techniques have been applied in smart learning. It shows the current state of the art by reviewing the main publications, the main type of fused educational data, and the data fusion approaches and techniques used in EDM/LA, as well as the main open problems, trends, and challenges in this specific research area.

show abstract

“…For the DEAP and DREAMER datasets, class imbalance occurred when AroVal scores were divided into Low and High classes by setting the threshold at the center of the scale. The synthetic minority oversampling technique (SMOTE) [47] addresses this problem by generating new data points from minority classes, and has been used in applications with sparse [48] and noisy [49] samples and multimodal physiological signals for affect classification [50] [51] [52]. In some cases [50] [51], models trained with SMOTE-generated data did not perform well due to the socalled "cold start" problem.…”

Section: Explainable Affect Recognitionmentioning

confidence: 99%

Emotion Recognition Using Explainable Genetically Optimized Fuzzy ART Ensembles

2021

View full text Add to dashboard Cite

There is a growing demand for explainability in complex artificial intelligence solutions to support critical applications' decision-making processes. Barriers to explainable processes include blackbox classifiers, such as deep learning, and noisy datasets. Affect recognition involving neural networks attempts to map complex human emotions onto Arousal and Valence scales based on physiological signal measurements. Datasets collected for this purpose are inherently noisy and may contain outliers and imbalanced classes, hindering accurate classification. In our approach, these issues are addressed using Fuzzy ART (FA) for clustering data samples into more condensed memory templates, introducing stochastic resonant noise to amplify signal-to-noise ratio, and SMOTE sampling to generate synthetic minority samples. A genetic algorithm is developed for FA optimization and ensemble model selection. Clusters obtained from the resulting ensembles are then used to train an ensemble of boosted decision trees for classification and to visualize the decision-making processes. Individual features such as heart rate variability and EEG band power, as well as feature interactions between pairs of features, may contain critical information as human affect indicators. Contributions of individual features and feature interactions toward describing human affect are quantified and interpreted using Shapley additive explanation values. Three established affect recognition datasets were considered for mapping physiological features onto a binary classification of Low/High Arousal and Positive/Negative Valence. Our framework was able to achieve good generalization for both classification tasks as well as provide detailed insights into the contributions of physiological features towards describing Arousal and Valence affects.

show abstract

Improving Affect Detection in Game-Based Learning with Multimodal Data Fusion

Cited by 27 publications

References 14 publications

Multimodal Classification: Current Landscape, Taxonomy and Future Directions

Multimodal Classification: Current Landscape, Taxonomy and Future Directions

A review on data fusion in multimodal learning analytics and educational data mining

Emotion Recognition Using Explainable Genetically Optimized Fuzzy ART Ensembles

Contact Info

Product

Resources

About