Rehabilitation of Traumatic Brain Injured Patients: Patient Mood Analysis from Multimodal Video

Abstract

Rehabilitation after traumatic brain injury (TBI) is very critical as it is largely unpredictable depending upon the nature of the injury. Rehabilitation process and recovery time also varies, as it takes months and years, depending upon the assessment of treatment, mental and physical conditions and strategies. Due to non-cooperative behaviour of patients, and increase in negative emotional expressions it is very beneficial to evaluate these expressions in a contactless way, and perform a rehabilitation physiotherapy, cognitive or other behavioral activities when the patient is in a positive mood. In this paper we have analyzed the methods for facial features extraction for TBI patients to determine optimal time to have aforementioned rehabilitation process on the basis of positive and negative facial expressions. We have employed a deep learning architecture based on convolutional neural network and long short term memory on RGB and thermal data that were collected in challenging scenarios from real patients. It automatically identifies the patient's facial expressions, and inform experts or trainers that "it is the time" to start rehabilitation session.

Teaching Pepper Robot to Recognize Emotions of Traumatic Brain Injured Patients Using Deep Neural Networks

“…The main aim of this study is to perform facial expression (FE) and mood recognition of TBI patients in order to enhance the social interaction and assist trainers and physiotherapists with the help of robots. First we accumulated a database in three uniform scenarios namely cognitive, physio and social rehabilitation activities, ensuring the reliability of the database as explained in detail in [13], [14]. This database is comprised of 924 videos taken about 11 participants, each being a maximum of 5 second in length, recorded with an Axis RGB and a Logitech RGB camera during multiple sessions at 30fps, resulting in approximately 140,000 captured frames.…”

“…Furthermore, Pepper identifies neutral expressions of TBI patients with only 42% accuracy with sad and neutral expression overlapping, proving that TBI patients' neutral expressions are more likely recognized as sad ones. Experts have annotated the patients' expressions as neutral since their ability to display emotional signals is disturbed due to injury, and during post stroke rehabilitation they exhibit depression and negative emotions more often than positive ones [13], [14]. It is also observed that the Pepper robot failed to identify fatigue expressions due to technical limitations.…”

“…These include the patients' non-cooperative behavior, inappropriate responses and inability to express their emotions. This is due to the nature of the condition like stroke or accident, resulting in damaged sensory motor control and reasoning skills, along with restricted muscle movements due to paralysis [13], [14]. However, these challenges can be different from patient-to-patient, depending on the nature and severity of the injury, producing speech inhibition, partial or complete paralysis, involuntary body movements, abrupt emotional changes, aggression, lack of consciousness or attention and varied emotion elicitation [15].…”

“…In addition to that, remarkable achievements has witnessed in machines analysis of human emotions, but there are still noticeable challenges that are needed to be addressed in order to involve robots into daily interfaces like social, physical or cognitive activities in real-world scenarios. Some of the major challenges are as follows: • Currently available datasets have FE of healthy people who are mostly cooperative and sometimes produce induced-expressions as compared to TBI patients who have impaired skills, and quite varied and limited expressions due to facial paralysis [13], [14]. Additionally, induced FEs that are produced consciously largely alter from natural involuntary emotions.…”

Developing a User-Centred Communication Pad for Cognitive and Physical Impaired People

“…As these residents are unable to recover from their life-altering impairments fully, the centre provides full-time care to them and aid in organizing and supporting activities of daily living (ADL). The project collaboration aim is to investigate where technological innovation can assist residents and staff members with fulfillment of rehabilitation activities -including enhancing individual self-control and improvement of quality of life [5,6,16].…”