Pain-related emotions are a major barrier to effective self rehabilitation in chronic pain. Automated coaching systems capable of detecting these emotions are a potential solution. This paper lays the foundation for the development of such systems by making three contributions. First, through literature reviews, an overview of how pain is expressed in chronic pain and the motivation for detecting it in physical rehabilitation is provided. Second, a fully labelled multimodal dataset (named ‘EmoPain’) containing high resolution multiple-view face videos, head mounted and room audio signals, full body 3D motion capture and electromyographic signals from back muscles is supplied. Natural unconstrained pain related facial expressions and body movement behaviours were elicited from people with chronic pain carrying out physical exercises. Both instructed and non-instructed exercises were considered to reflect traditional scenarios of physiotherapist directed therapy and home-based self-directed therapy. Two sets of labels were assigned: level of pain from facial expressions annotated by eight raters and the occurrence of six pain-related body behaviours segmented by four experts. Third, through exploratory experiments grounded in the data, the factors and challenges in the automated recognition of such expressions and behaviour are described, the paper concludes by discussing potential avenues in the context of these findings also highlighting differences for the two exercise scenarios addressed.

Human behavior is the focus of many studies in the social, health, and behavioral sciences. Yet, few studies use behavioral observation methods to collect objective measures of behavior as it occurs in daily life, out in the real worldpresumably the context of ultimate interest. Here we provide a review of recent studies focused on measuring human behavior using smartphones and their embedded mobile sensors. To draw attention to current advances in the field of smartphone sensing, we describe the daily behaviors captured using these methods, which include movement behaviors (physical activity, mobility patterns), social behaviors (faceto-face encounters, computer-mediated communications), and other daily activities (nonmediated and mediated activities). We conclude by pointing to promising areas of future research for studies using Smartphone Sensing Methods (SSMs) in the behavioral sciences.

Objective This purpose of this study was to describe and demonstrate CrossCheck, a multimodal data collection system designed to aid in continuous remote monitoring and identification of subjective and objective indicators of psychotic relapse. Methods Individuals with schizophrenia-spectrum disorders received a smartphone with the monitoring system installed along with unlimited data plan for 12 months. Participants were instructed to carry the device with them and to complete brief self-reports multiple times a week. Multi-modal behavioral sensing (i.e., physical activity, geospatial activity, speech frequency and duration) and device use data (i.e., call and text activity, app use) were captured automatically. Five individuals who experienced psychiatric hospitalization were selected and described for instructive purposes. Results Participants had unique digital indicators of their psychotic relapse. For some, self-reports provided clear and potentially actionable description of symptom exacerbation prior to hospitalization. Others had behavioral sensing data trends (e.g., shifts in geolocation patterns, declines in physical activity) or device use patterns (e.g., increased nighttime app use, discontinuation of all smartphone use) that reflected the changes they experienced more effectively. Conclusion Advancements in mobile technology are enabling collection of an abundance of information that until recently was largely inaccessible to clinical research and practice. However, remote monitoring and relapse detection is in its nascency. Development and evaluation of innovative data management, modeling, and signal-detection techniques that can identify changes within an individual over time (i.e. unique relapse signatures) will be essential if we are to capitalize on these data to improve treatment and prevention.

Continuously monitoring schizophrenia patients’ psychiatric symptoms is crucial for in-time intervention and treatment adjustment. The Brief Psychiatric Rating Scale (BPRS) is a survey administered by clinicians to evaluate symptom severity in schizophrenia. The CrossCheck symptom prediction system is capable of tracking schizophrenia symptoms based on BPRS using passive sensing from mobile phones. We present results from an ongoing randomized control trial, where passive sensing data, self-reports, and clinician administered 7-item BPRS surveys are collected from 36 outpatients with schizophrenia recently discharged from hospital over a period ranging from 2-12 months. We show that our system can predict a symptom scale score based on a 7-item BPRS within ±1.45 error on average using automatically tracked behavioral features from phones (e.g., mobility, conversation, activity, smartphone usage, the ambient acoustic environment) and user supplied self-reports. Importantly, we show our system is also capable of predicting an individual BPRS score within ±1.59 error purely based on passive sensing from phones without any self-reported information from outpatients. Finally, we discuss how well our predictive system reflects symptoms experienced by patients by reviewing a number of case studies.