UbiEar

Abstract

Non-speech sound-awareness is important to improve the quality of life for the deaf and hard-of-hearing (DHH) people. DHH people, especially the young, are not always satisfied with their hearing aids. According to the interviews with 60 young hard-of-hearing students, a ubiquitous sound-awareness tool for emergency and social events that works in diverse environments is desired. In this paper, we design UbiEar, a smartphone-based acoustic event sensing and notification system. Core techniques in UbiEar are a light-weight deep convolution neural network to enable location-independent acoustic event recognition on commodity smartphons, and a set of mechanisms for prompt and energy-efficient acoustic sensing. We conducted both controlled experiments and user studies with 86 DHH students and showed that UbiEar can assist the young DHH students in awareness of important acoustic events in their daily life.

Deaf and Hard-of-hearing Individuals' Preferences for Wearable and Mobile Sound Awareness Technologies

“…Our findings show that while most participants were "very" or "extremely" interested in being aware of sounds, this interest was modulated by communication preference-that is, preference for sign or oral communication or both. We provide a prioritization of sound types and sound characteristics (e.g., urgent alerts and voices directed at you are considered of highest interest), which extends past work [3,27,39] with quantitative and/or more detailed results. Further, we find that smartwatches are the most preferred individual form factor, but 92% of participants wanted both visual and haptic feedback and 75% of participants wanted that feedback to be provided on separate devices (haptic on smartwatch, visual on smartphone or HMD).…”

“…Early work by Matthews et al [27] examined sound awareness needs across a variety of contexts (at home, at work, while mobile), and built and evaluated a proof-of-concept prototype to display office sounds on a computer monitor. More recent solutions have begun to investigate other form factors, including head-mounted displays for speech captioning [16,17,34], wrist-worn or smartwatch displays [20,29], and smartphone apps for general sound detection [3,30,39]. While formative studies and, in some cases, evaluations of these technologies have yielded useful insights, the studies tend to be qualitative and/or focused on a single device form factor and have not been designed to examine issues around social acceptability.…”

“…The contributions of this paper include: (1) extension of findings on sound awareness interest [3,27,39] to include influences of demographic factors; (2) characterization of preferences for form factor, output modality, and speech vs. non-speech sounds; (3) comparison of subjective utility and social comfort related to these form factors and feedback modalities; (4) design recommendations for future mobile or wearable sound awareness technologies.…”

Use of Machine Learning by Non-Expert DHH People: Technological Understanding and Sound Perception

“…Unlike previous approaches to support sensory disabilities without ML including tactile devices, sensor networks [7,34,37], ML-based sensors can convey more complex information to users, and can potentially handle diverse demands from users. For example, ML-based speech recognition can be used to support DHH people [26,38], and another prior study used ML to recognize environmental sound around DHH users [33]. However, despite the advances in ML-based sound recognition technologies [6,14], it is still difficult to train a single recognition model to meet all requirements from diverse users.…”

“…and sound recognition technologies, for example, can help visuallyimpaired and deaf and hard-of-hearing (DHH) people by providing additional sensory modalities. Hearing loss is known as one of the most common disability, and some existing studies have investigated the potential of ML-based assistive technology to support DHH people in some specific scenarios, such as detecting sounds of ringing doorbells and crying people [4,33]. However, compared to the case of visual recognition, it is more difficult to support DHH people with ML because of the difficulty of representing/visualizing audio data.…”