What cognitive and affective states should technology monitor to support learning?

Olugbade, Temitayo; Cuturi, Luigi F.; Cappagli, Giulia; Volta, Erica; Alborno, Paolo; Newbold, Joseph W.; Bianchi-Berthouze, Nadia; Baud-Bovy, Gabriel; Volpe, Gualtiero; Gori, Monica

doi:10.1145/3139513.3139522

Cited by 6 publications

(5 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Findings in [5] suggest that this cognitive strategy may in fact be necessary to enable solution of mathematical problems. This understanding higlights the need for digital learning technology to create the setting for a learner to apply reflecting thinking strategies [6]. Indeed, the possibility of guiding a learner through the use of reflective thinking in solving mathematical problems was demonstrated in [5].…”

Section: Introductionmentioning

confidence: 94%

See 1 more Smart Citation

Automatic Detection of Reflective Thinking in Mathematical Problem Solving Based on Unconstrained Bodily Exploration

Olugbade

Newbold

Johnson

et al. 2022

IEEE Trans. Affective Comput.

Self Cite

View full text Add to dashboard Cite

For technology (like serious games) that aims to deliver interactive learning, it is important to address relevant mental experiences such as reflective thinking during problem solving. To facilitate research in this direction, we present the weDraw-1 Movement Dataset of body movement sensor data and reflective thinking labels for 26 children solving mathematical problems in unconstrained settings where the body (full or parts) was required to explore these problems. Further, we provide qualitative analysis of behaviours that observers used in identifying reflective thinking moments in these sessions. The body movement cues from our compilation informed features that lead to average F1 score of 0.73 for automatic detection of reflective thinking based on Long Short-Term Memory neural networks. We further obtained 0.79 average F1 score for end-toend detection of reflective thinking periods, i.e. based on raw sensor data. Finally, the algorithms resulted in 0.64 average F1 score for period subsegments as short as 4 seconds. Overall, our results show the possibility of detecting reflective thinking moments from body movement behaviours of a child exploring mathematical concepts bodily, such as within serious game play.

show abstract

Section: Introductionmentioning

confidence: 94%

“…However, the convolution function could be useful in integrating temporal and postural patterns, similar to the LSTM, albeit only locally. The Positional energy of the head 5 , left and right hand 6,7 and knee 8,9…”

Section: Generalisability Of Other Standard Algorithmsmentioning

confidence: 99%

Automatic Detection of Reflective Thinking in Mathematical Problem Solving Based on Unconstrained Bodily Exploration

Olugbade

Newbold

Johnson

et al. 2022

IEEE Trans. Affective Comput.

Self Cite

View full text Add to dashboard Cite

show abstract

“…A sign of excitement and curiosity may indicate that the child is ready to be challenged, while over-confidence may lead a child to overlook details (for a review, see Olugbade et al, 2017).…”

Section: Developmentmentioning

confidence: 99%

Multisensory Perception and Learning: Linking Pedagogy, Psychophysics, and Human–Computer Interaction

et al. 2022

Self Cite

View full text Add to dashboard Cite

In this review, we discuss how specific sensory channels can mediate the learning of properties of the environment. In recent years, schools have increasingly been using multisensory technology for teaching. However, it still needs to be sufficiently grounded in neuroscientific and pedagogical evidence. Researchers have recently renewed understanding around the role of communication between sensory modalities during development. In the current review, we outline four principles that will aid technological development based on theoretical models of multisensory development and embodiment to foster in-depth, perceptual, and conceptual learning of mathematics. We also discuss how a multidisciplinary approach offers a unique contribution to development of new practical solutions for learning in school. Scientists, engineers, and pedagogical experts offer their interdisciplinary points of view on this topic. At the end of the review, we present our results, showing that one can use multiple sensory inputs and sensorimotor associations in multisensory technology to improve the discrimination of angles, but also possibly for educational purposes. Finally, we present an application, the ‘RobotAngle’ developed for primary (i.e., elementary) school children, which uses sounds and body movements to learn about angles.

show abstract

“…Thus, digital learning technology that promotes the use of body movement as well as further recognizes critical learning moments (e.g. reflective thinking) could support learning of abstract mathematical ideas which may otherwise be challenging to relate to [35].…”

Section: B Detection Of Reflective Thinking Based On Body Movement Datamentioning

confidence: 99%

The AffectMove 2021 Challenge - Affect Recognition from Naturalistic Movement Data

Olugbade

Sagoleo

Ghisio

et al. 2021

2021 9th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW)

Self Cite

View full text Add to dashboard Cite

We ran the first Affective Movement Recognition (AffectMove) challenge that brings together datasets of affective bodily behaviour across different real-life applications to foster work in this area. Research on automatic detection of naturalistic affective body expressions is still lagging behind detection based on other modalities whereas movement behaviour modelling is a very interesting and very relevant research problem for the affective computing community. The AffectMove challenge aimed to take advantage of existing body movement datasets to address key research problems of automatic recognition of naturalistic and complex affective behaviour from this type of data. Participating teams competed to solve at least one of three tasks based on datasets of different sensors types and reallife problems: multimodal EmoPain dataset for chronic pain physical rehabilitation context, weDraw-1 Movement dataset for maths problem solving settings, and multimodal Unige-Maastricht Dance dataset. To foster work across datasets, we also challenged participants to take advantage of the data across datasets to improve performances and also test the generalization of their approach across different applications.

show abstract

What cognitive and affective states should technology monitor to support learning?

Cited by 6 publications

References 25 publications

Automatic Detection of Reflective Thinking in Mathematical Problem Solving Based on Unconstrained Bodily Exploration

Automatic Detection of Reflective Thinking in Mathematical Problem Solving Based on Unconstrained Bodily Exploration

Multisensory Perception and Learning: Linking Pedagogy, Psychophysics, and Human–Computer Interaction

The AffectMove 2021 Challenge - Affect Recognition from Naturalistic Movement Data

Contact Info

Product

Resources

About