Facial and vocal markers of schizophrenia measured using remote smartphone assessments

Abstract

BackgroundMachine learning-based facial and vocal measurements have demonstrated relationships with schizophrenia diagnosis and severity. Here, we determine their accuracy of when acquired through automated assessments conducted remotely through smartphones. Demonstrating utility and validity of remote and automated assessments conducted outside of controlled experimental settings can facilitate scaling such measurement tools to aid in risk assessment and tracking of treatment response in difficult to engage populations.MethodsMeasurements of facial and vocal characteristics including facial expressivity, vocal acoustics, and speech prevalence were assessed in 20 schizophrenia patients over the course of 2 weeks in response to two classes of prompts previously utilized in experimental laboratory assessments: evoked prompts, where subjects are guided to produce specific facial expressions and phonations, and spontaneous prompts, where subjects are presented stimuli in the form of emotionally evocative imagery and asked to freely respond. Facial and vocal measurements were assessed in relation to schizophrenia symptom severity using the Positive and Negative Syndrome Scale.ResultsVocal markers including speech prevalence, vocal jitter, fundamental frequency, and vocal intensity demonstrated specificity as markers of negative symptom severity while measurement of facial expressivity demonstrated itself as a robust marker of overall schizophrenia severity.ConclusionEstablished facial and vocal measurements, collected remotely in schizophrenia patients via smartphones in response to automated task prompts, demonstrated accuracy as markers of schizophrenia severity. Clinical implications are discussed.

Validation of Visual and Auditory Digital Markers of Suicidality in Acutely Suicidal Psychiatric Inpatients: Proof-of-Concept Study

“…For example, virtual clinic visits through video calls can involve digital measurements to assess mental health if such measurements are integrated into the technology utilized. Existing smartphone-based platforms allow for the collection of video and audio data and subsequent calculation of digital measures, and can be applied in out-patient settings to remotely acquire these measurements without adding further burden on the clinical staff [36][37][38]. Of course, such technologies face a great deal of challenges before they can be deployed in patient care, starting with further validation of methodologies, passage through regulatory review as dependable clinical measures, and adoption by health care professionals into their day-to-day clinical workflows [39].…”

Remote Digital Measurement of Facial and Vocal Markers of Major Depressive Disorder Severity and Treatment Response: A Pilot Study

“…This software platform has historically been used in clinical research for reporting of patient behavior to clinicians, including medication adherence, electronic patient-reported outcomes, and ecological momentary assessments, with considerable work done on patient acceptance and usability ( 32 , 33 ). An additional functionality of capturing video and audio in response to prompts (as described below) was utilized for the purposes of this study ( 34 , 35 ).…”

Digital Phenotyping in Livestock Farming

“…Sound-based data sensors can be attached to individual animals for sound analysis. As demonstrated by human studies, acoustic sensor data can be refined and analyzed to predict emotional states and some actions [49,50]. It is not unreasonable to expect the same to be true for animals due to their abundant vocalizations that are specific to stress, breeding, eating, and socialization.…”