Early Prediction of Visitor Engagement in Science Museums with Multimodal Learning Analytics

Emerson, Andrew; Henderson, Nathan; Rowe, Jonathan P.; Min, Wookhee; Lee, Seung; Minogue, James; Lester, James C.

doi:10.1145/3382507.3418890

Cited by 19 publications

(11 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More from the teachers perspective, ML was used to process multimodal data to identify teachers' key instructional segments to assess their instruction (Donnelly et al, 2016) and to extract orchestration graphs including teaching activities and their social plane over time (Prieto et al, 2018). In addition, other studies focused on engagement to predict efficacious learning practices , in the contexts of museums (Emerson, Henderson, et al, 2020); the context of an adaptive assessment platform (Sharma, Papamitsiou, et al, 2020) and learning with the internet of things (Camacho et al, 2020). Proposing MMLA prototypes without specific evidence on impact: The search pointed to twenty-two studies that proposed MMLA prototypes, yet did not provide any specific evidence about the impact of the system on the expected educational outcomes.…”

Section: Existing Evidence On the Use Of Mmla To Support Educational Outcomesmentioning

confidence: 99%

The evidence of impact and ethical considerations of Multimodal Learning Analytics: A Systematic Literature Review

Alwahaby¹,

Cukurova²,

Papamitsiou³

et al. 2021

Preprint

View full text Add to dashboard Cite

There is a growing interest in the research and use of multimodal data in learning analytics. This paper presents a systematic literature review of multimodal learning analytics (MMLA) research to assess i) the available evidence of impact on learning outcomes in real-world contexts and ii) explore the extent to which ethical considerations are addressed. A few recent literature reviews argue for the promising value of multimodal data in learning analytics research. However, our understanding of the challenges associated with MMLA research from real-world teaching and learning environments is limited. To address this gap, this paper provides an overview of the evidence of impact and ethical considerations stemming from an analysis of the relevant MMLA research published in the last decade. The search of the literature resulted in 663 papers, of which 100 were included in the final synthesis. The results show that the evidence of real-world impact on learning outcomes is weak, and ethical aspects of MMLA work are rarely addressed. We discuss our results through the lenses of two theoretical frameworks for evidence of impact types and ethical dimensions of MMLA. We conclude that for MMLA to stay relevant and become part of mainstream education, future research should directly address the gaps identified in this review.

show abstract

Section: Existing Evidence On the Use Of Mmla To Support Educational Outcomesmentioning

confidence: 99%

The evidence of impact and ethical considerations of Multimodal Learning Analytics: A Systematic Literature Review

Alwahaby¹,

Cukurova²,

Papamitsiou³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…While formal and informal learning environments share many core objectives, museum-based learning presents distinctive challenges for measuring learner engagement, including short dwell times and an emphasis on free-choice learning. Well-designed, engaging exhibits frequently have very brief visitor interactions, and even highly engaged visitors can have short dwell times [17]. Furthermore, museums often attract a diverse range of visitors in terms of age, gender, and socio-cultural background.…”

Section: Museum Visitor Engagementmentioning

confidence: 99%

“…Future Worlds is a prototype game-based museum exhibit about environmental sustainability [17]. In Future Worlds, visitors interact with a touch-based display to learn about environmental sustainability by investigating the impacts of alternative decisions made within a 3D virtual environment.…”

Section: Future Worlds Testbed Exhibitmentioning

confidence: 99%

“…Several predictive features were extracted from the collected data based on prior work using multimodal learning analytics to predict museum visitor engagement [17]. Eight features were distilled from the interaction trace log data, including the total number of times that a visitor tapped on the interactive display and the total number of times a visitor tapped to examine informational texts about several in-game elements.…”

Section: Multimodal Featuresmentioning

confidence: 99%

“…Recent advances in multimodal learning analytics have been used to detect patterns of learner engagement by utilizing multiple sensor-based data streams such as facial expression, posture, eye gaze, and interaction log data [3,14,28,39]. Although the benefits of incorporating multimodal data streams into models of learner engagement has been demonstrated in both classroom and laboratory settings [13,27,28,31], investigating multimodal analytics in informal settings is still in its early stages [17].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

What's Fair is Fair: Detecting and Mitigating Encoded Bias in Multimodal Models of Museum Visitor Attention

Acosta

Henderson

Rowe

et al. 2021

Proceedings of the 2021 International Conference on Multimodal Interaction

Self Cite

View full text Add to dashboard Cite

Recent years have seen growing interest in modeling visitor engagement in museums with multimodal learning analytics. In parallel, there has also been growing concern about issues of fairness and encoded bias in machine learning models. In this paper, we investigate bias detection and mitigation techniques to address issues of algorithmic fairness in multimodal models of museum visitor visual attention. We employ slicing analysis using the Absolute Between-ROC Area (ABROCA) statistic to detect encoded bias present in multimodal models of visitor visual attention trained with facial expression and posture data from visitor interactions with a game-based museum exhibit about environmental sustainability. We investigate instances of gender bias that arise between different combinations of modalities across several machine learning techniques. We also measure the effectiveness of two different debiasing strategies-learned fair representations and reweighing-when applied to the trained multimodal visitor attention models. Results indicate that patterns of bias can arise across different modality combinations for the different visitor visual attention models, and there is often an inherent tradeoff between predictive accuracy and ABROCA. Analyses suggest that debiasing strategies tend to be more effective on multimodal models of visitor visual attention than their unimodal counterparts CCS CONCEPTS• Applied computing → Education; • Computing methodologies → Machine Learning.

show abstract

Multimodal Trajectory Analysis of Visitor Engagement with Interactive Science Museum Exhibits

Emerson

Henderson

Min

et al. 2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Early Prediction of Visitor Engagement in Science Museums with Multimodal Learning Analytics

Cited by 19 publications

References 35 publications

The evidence of impact and ethical considerations of Multimodal Learning Analytics: A Systematic Literature Review

The evidence of impact and ethical considerations of Multimodal Learning Analytics: A Systematic Literature Review

What's Fair is Fair: Detecting and Mitigating Encoded Bias in Multimodal Models of Museum Visitor Attention

Multimodal Trajectory Analysis of Visitor Engagement with Interactive Science Museum Exhibits

Contact Info

Product

Resources

About