Automatic Recognition of Multiple Affective States in Virtual Rehabilitation by Exploiting the Dependency Relationships

Rivas, Jesús Joel; Orihuela‐Espina, Felipe; Sucar, Luis Enrique; Williams, Amanda C de C; Bianchi-Berthouze, Nadia

doi:10.1109/acii.2019.8925508

Cited by 9 publications

(10 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In their work, the authors explore the use of hand pressure and joystick manipulation to detect stroke patients' pain level by personalising the model to each patient by using data from 10 different sessions. In [39], the authors extend the work by combining multiple modalities (hand pressure, gesture and facial expressions) to investigate the relationship between affective states and pain during rehabilitation. Again, individual models are built by taking advantage of the multiple sessions.…”

Section: B Automatic Pain Detection Based On Bodily Expressionsmentioning

confidence: 99%

EMOPAIN Challenge 2020: Multimodal Pain Evaluation from Facial and Bodily Expressions

Egede

Song

Olugbade

et al. 2020

2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020)

Self Cite

View full text Add to dashboard Cite

The EmoPain 2020 Challenge is the first international competition aimed at creating a uniform platform for the comparison of multi-modal machine learning and multimedia processing methods of chronic pain assessment from human expressive behaviour, and also the identification of pain-related behaviours. The objective of the challenge is to promote research in the development of assistive technologies that help improve the quality of life for people with chronic pain via real-time monitoring and feedback to help manage their condition and remain physically active. The challenge also aims to encourage the use of the relatively underutilised, albeit vital bodily expression signals for automatic pain and painrelated emotion recognition. This paper presents a description of the challenge, competition guidelines, bench-marking dataset, and the baseline systems' architecture and performance on the Challenge's three sub-tasks: pain estimation from facial expressions, pain recognition from multimodal movement, and protective movement behaviour detection.

show abstract

Section: B Automatic Pain Detection Based On Bodily Expressionsmentioning

confidence: 99%

EMOPAIN Challenge 2020: Multimodal Pain Evaluation from Facial and Bodily Expressions

Egede

Song

Olugbade

et al. 2020

2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020)

Self Cite

View full text Add to dashboard Cite

show abstract

“…1) a novel architecture formed by a multi-label classifier called Circular Classifier Chain (CCC) combined with a set of multimodal classifiers called Fusion 1. This work is an extension of our research presented at the International Conference of Affective Computing and Intelligent Interaction ACII 2019 [11]. In this extended version, we included experimental validation by incrementing the number of post-stroke patients (from five to eight) in the longitudinal study; we extended the analysis of the convergence of the proposed multi-label classifier, Circular Classifier Chain (CCC); and we did further analysis to evaluate the impact of the affective states ordering within the CCC.…”

Section: Introductionmentioning

confidence: 91%

“…The first version of this dataset, including 5 patients, was presented in [14], where only finger pressure and hand movements data were used. The dataset was then used in [11] by considering all modalities available, finger pressure, hand movements, and facial expressions. For this current paper, the dataset was extended to include 6 new patients, as described below.…”

Section: Dataset Of Spontaneous Affective States Of Post-stroke Patientsmentioning

confidence: 99%

Multi-Label and Multimodal Classifier for Affective States Recognition in Virtual Rehabilitation

Rivas

Lara

Castrejón

et al. 2022

IEEE Trans. Affective Comput.

Self Cite

View full text Add to dashboard Cite

Computational systems that process multiple affective states may benefit from explicitly considering the interaction between the states to enhance their recognition performance. This work proposes the combination of a multi-label classifier, Circular Classifier Chain (CCC), with a multimodal classifier, Fusion using a Semi-Naive Bayesian classifier (FSNBC), to include explicitly the dependencies between multiple affective states during the automatic recognition process. This combination of classifiers is applied to a virtual rehabilitation context of post-stroke patients. We collected data from post-stroke patients, which include finger pressure, hand movements, and facial expressions during ten longitudinal sessions. Videos of the sessions were labelled by clinicians to recognize four states: tiredness, anxiety, pain, and engagement. Each state was modelled by the FSNBC receiving the information of finger pressure, hand movements, and facial expressions. The four FSNBCs were linked in the CCC to exploit the dependency relationships between the states. The convergence of CCC was reached by 5 iterations at most for all the patients. Results (ROC AUC) of CCC with the FSNBC are over 0.940 ± 0.045 (mean ± std. deviation) for the four states. Relationships of mutual exclusion between engagement and all the other states and co-occurrences between pain and anxiety were detected and discussed.

show abstract

“…during a consultation interview with a therapist [25]) or the type of activity is constant throughout (e.g. the detection of pain and anxiety in game-based physical rehabilitation [53,71]). Bodily expression is also used to inform healthcare applications, e.g.…”

Section: Affective Movement Behavior Detectionmentioning

confidence: 99%

Leveraging Activity Recognition to Enable Protective Behavior Detection in Continuous Data

Wang

Gao

Mathur

et al. 2021

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

Self Cite

View full text Add to dashboard Cite

Protective behavior exhibited by people with chronic pain (CP) during physical activities is very informative to understanding their physical and emotional states. Existing automatic protective behavior detection (PBD) methods rely on pre-segmentation of activities predefined by users. However, in real life, people perform activities casually. Therefore, where those activities present difficulties for people with CP, technology-enabled support should be delivered continuously and automatically adapted to activity type and occurrence of protective behavior. Hence, to facilitate ubiquitous CP management, it becomes critical to enable accurate PBD over continuous data. In this paper, we propose to integrate human activity recognition (HAR) with PBD via a novel hierarchical HAR-PBD architecture comprising graph-convolution and long short-term memory (GC-LSTM) networks, and alleviate class imbalances using a class-balanced focal categorical cross-entropy (CFCC) loss. Through in-depth evaluation of the approach using a CP patients' dataset, we show that the leveraging of HAR, GC-LSTM networks, and CFCC loss leads to clear increase in PBD performance against the baseline (macro F1 score of 0.81 vs. 0.66 and precision-recall area-under-the-curve (PR-AUC) of 0.60 vs. 0.44). We conclude by discussing possible use cases of the hierarchical architecture in CP management and beyond. We also discuss current limitations and ways forward.

show abstract

Automatic Recognition of Multiple Affective States in Virtual Rehabilitation by Exploiting the Dependency Relationships

Cited by 9 publications

References 23 publications

EMOPAIN Challenge 2020: Multimodal Pain Evaluation from Facial and Bodily Expressions

EMOPAIN Challenge 2020: Multimodal Pain Evaluation from Facial and Bodily Expressions

Multi-Label and Multimodal Classifier for Affective States Recognition in Virtual Rehabilitation

Leveraging Activity Recognition to Enable Protective Behavior Detection in Continuous Data

Contact Info

Product

Resources

About