Use of Machine Learning by Non-Expert DHH People: Technological Understanding and Sound Perception

Abstract

Recent advances in machine learning demonstrated its potential in accessibility applications. However, recognition models and their application scenarios are often defined by machine learning (ML) experts and cannot fully capture users' diverse demands with disabilities. In order to open up the full potential of ML for accessibility applications, we have to bridge the gap for non-expert people doubly caused by the technological understanding and their disabilities. In this work, we investigate how non-expert deaf and hard-of-hearing (DHH) people understand ML technologies and design ML-based sound recognition systems. We conduct a workshop study consisting of an ML lecture and an interactive learning session using a sound recognition system. Through observations during the workshop and semi-structured interviews, we clarify that non-expert DHH people start to overcome the knowledge gap. They could obtain a more detailed understanding of ML technology and how to use sounds to train ML models. CCS CONCEPTS • Social and professional topics → People with disabilities; • Computing methodologies → Machine learning.

Toward User-Driven Sound Recognizer Personalization with People Who Are d/Deaf or Hard of Hearing

“…This study confrms the potential for non-expert DHH users to train personalizable sound recognizers (as identifed in past work [58]) and advances understanding of: (1) how non-expert DHH users approach in situ recording tasks to create a sound recognizer training set, (2) practical challenges that they may face when recording a variety real-world sounds, and (3) sense-making strategies that they use to interpret audio data in this context. Here, we discuss implications of our fndings, opportunities for future work, and the limitations of our paper.…”

“…Personalization is essential to provide context-specifc support and meet the wide-ranging needs of the DHH community [22,27,53,55,58]. Some projects have explored options for DHH users to flter notifcations for certain sounds [27,37,38], but they stopped short of adding or modifying sound classes through user-provided recordings.…”

“…For example, a blind or visually impaired user is likely unable to use visual feedback when capturing images for personalizing an object recognizer-a challenge that Kacorri et al and others (e.g., [41,69]) frst examined via studies of users' needs in this context, and more recently began addressing through active feedback techniques to assist in image capture [49]. Indeed, in Nakao et al's [58] study of DHH users' technical understanding of ML, workshop participants struggled to choose acceptable sound samples for training due to a lack of non-auditory feedback. For work on sampling feedback, VoiceAssist [65] provides real-time visual feedback to help inexperienced users reduce reverberation and background noise in voice recordings, and a user study showed third party listeners preferred recordings made using VoiceAssist compared to those without.…”

“…Google's Teachable Machine experiment [11,28] allows non-expert users to quickly train a personal sound recognition model with their own audio samples, but ML expertise is required to have agency over the produced model (e.g., redefning features). Nakao et al [58] studied non-expert DHH users' understanding of ML with a similar interactive workfow that supported training sets built from users' recordings or selected from a large sound library [26]. DHH participants overcame gaps in their technical understanding and identifed additional use cases after hands-on experience, but some struggled to review samples and defne ML tasks for sounds they knew but could not hear.…”

“…Participants also considered the diversity of samples within each sound class-common among non-experts (e.g., [33,58,78]). Many decided to limit diversity by producing the sound the same way in each sample: "I want the sounds to be relatively consistent, just so the machine learning device isn't like, 'You have three diferent weird noises, but you say they're all the same'" (P4).…”