Crowd++

Abstract

Smartphones are excellent mobile sensing platforms, with the microphone in particular being exercised in several audio inference applications. We take smartphone audio inference a step further and demonstrate for the first time that it's possible to accurately estimate the number of people talking in a certain place -with an average error distance of 1.5 speakers -through unsupervised machine learning analysis on audio segments captured by the smartphones. Inference occurs transparently to the user and no human intervention is needed to derive the classification model. Our results are based on the design, implementation, and evaluation of a system called Crowd++, involving 120 participants in 10 very different environments. We show that no dedicated external hardware or cumbersome supervised learning approaches are needed but only off-the-shelf smartphones used in a transparent manner. We believe our findings have profound implications in many research fields, including social sensing and personal wellbeing assessment.

COVID-19 and Computer Audition: An Overview on What Speech & Sound Analysis Could Contribute in the SARS-CoV-2 Corona Crisis

“…Speaker diarization, i.e., determining who is speaking when, and speaker counting (20) can become of interest in the ongoing social distancing. When it comes to counter measures such as quarantine, or risk assessment of individuals, one could also consider the usage of automatic recognition of deceptive speech when people are questioned about their recent contacts or whereabouts, as their personal work and life interests may interfere with the perspective of being sent to quarantine.…”

Computer-Cognition Interfaces: Sensing and influencing mental processes with computer interaction

“…[236] A prevalent research agenda within ubiquitous computing is sensing. Sensing refers to the activity of computationally inferring (often human) context from real life situations, such as assessing the amount of people in a room based on sounds [243], or predicting how tired a person is based on their phone activity [80]. Approaches vary in both sensing domains (e.g., physical activity, cognitive), sensor types (e.g., accelerometer, microphone), time frames (e.g., real-time, weeks), active or passive sensing (e.g., direct manipulation, background sensing), as well as in modeling approaches (e.g., correlation, regression, classification).…”

“…activity [38] device use [83] car speed [35] emotions [177] energy use [5] coughing [124] device position [79] device position [166] whereabouts [56] lung function [123] emotions [46] stress [134] location [163] dangerous driving [246] appliance usage [250] no. people [243] firefighters [63] walking [25] heart rate [213] transport mode [192] skin disease [77] sleep [80] car position [135] tooth brushing [114] running [88] academic performance [231] boredom [172] nursing activity [99] depression [33] alertness [1] generic [242] lung function [106] blood [229] emotions [149] mental health [230] alcohol [11] skin disease [137] blood [81] brain injury [137] emotions [143] engagement [171] heart rate [148] schizophrenia [232] whereabouts [223] app use …”

Low-resource Multi-task Audio Sensing for Mobile and Embedded Devices via Shared Deep Neural Network Representations

“…Audio sensing applications are cornerstone elements in mobile ubiquitous computing as evidenced by the rich array of behavioral insights they provide for mobile users. Examples are song recognition [4], speaker identification [38], emotion recognition [39,48], speaker counting [56], conversation analysis [34], voice commands [14], ambient sound analysis [40,42]. Until now modeling has focused primarily on discovering new sensing modalities or inference capabilities from human behavior rather than optimizing embedded resource use.…”