Opportunistic and Context-Aware Affect Sensing on Smartphones

Abstract

Opportunistic affect sensing offers unprecedented potential for capturing spontaneous affect, eliminating biases inherent in the controlled setting. Facial expression and voice are two major affective displays, however most affect sensing systems on smartphone avoid them due to extensive power requirements. Encouragingly, due to the recent advent of low-power DSP (Digital Signal Processing) co-processor and GPU (Graphics Processing Unit) technology, audio and video sensing are becoming more feasible on smartphone. To utilize opportunistically captured facial expression and voice, gathering contextual information about the dynamic audio-visual stimuli is also important. This paper discusses recent advances of affect sensing on the smartphone and identifies the key barriers and potential solutions for implementing opportunistic and contextaware affect sensing on smartphone platforms. In addition to exploring the technical challenges (privacy, battery life and robust algorithms), the challenges of recruiting and retention of mental health patients have also been considered; as experimentation with mental health patients is difficult but crucial to showcase the importance/effectiveness of the smartphone centred affect sensing technology. Affect SensingAffect sensing on smartphones is relatively new. Based on the sensing modalities, we group the existing studies into five categories: (1) phone usage (2) phone interaction (3) sensor feed (4) hybrid and (5) self-reporting.Phone Usage: includes communication patterns, i.e., statistics of call and text messages. It also includes the frequency of using social media, application usage patterns (for security reason no longer available to a non-system app), and switching between applications.

Ethical perspectives on recommending digital technology for patients with mental illness

“…This discussion only provides a limited list of the ethical challenges and does not offer specific solutions to these complex problems. Many significant issues were omitted such as how patient monitoring systems handle data that are inadvertently captured about other people such as facial images, voice recordings, and metadata (Rana et al 2016 ), and new legal issues such as timeliness of response to monitoring data (Armontrout et al 2016 ). Other issues that were omitted include whether health-related chatbots (automated conversational software) should deceive patients into thinking they are interacting with a human (Whitby 2014 ), the coming of medications with sensors for adherence monitoring (Kane et al 2013 ), the monitoring of people with dementia (Niemeijer et al 2011 ), and the evaluation of long-term clinical value.…”

Mobile activity recognition and fall detection system for elderly people using Ameva algorithm

“…Smartphones, which embed all these sensors, can be considered a type of wearable due to their pervasiveness. Furthermore, the low adoption barrier on healthcare applications [28] through application markets such as Google Play or AppStore makes them the best option to target the mass market. Some of them are focused on fall detection [29,30], but normally do not cover both ADL and falls [31], so a classification system must be designed to consider them.…”

Overview of context-sensitive technologies for well-being

“…Despite of being a complex and very CPU intensive process, the analysis of voice and facial expression can also be used as a monitoring mechanism [20]. It allows to get an accurate approximation about current well-being levels.…”

“…According to Rana, Margee, Reilly, Jurdak and Soar (2016), another factor almost never monitored during this kind of studies is opportunistic facial expression, being this a factor that allows inferring the current mood state with high accuracy [20].…”