Measuring Voice UX Quantitatively

Abstract

Computer voice is experiencing a renaissance through the growing popularity of voice-based interfaces, agents, and environments. Yet, how to measure the user experience (UX) of voice-based systems remains an open and urgent question, especially given that their form factors and interaction styles tend to be non-visual, intangible, and often considered disembodied or "body-less. " As a frst step, we surveyed the ACM and IEEE literatures to determine which quantitative measures and measurements have been deemed important for voice UX. Our fndings show that there is little consensus, even with similar situations and systems, as well as an overreliance on lab work and unvalidated scales. In response, we ofer two high-level descriptive frameworks for guiding future research, developing standardized instruments, and informing ongoing review work. Our work highlights the current strengths and weaknesses of voice UX research and charts a path towards measuring voice UX in a more comprehensive way. CCS CONCEPTS• Human-centered computing → Human computer interaction (HCI); Human computer interaction (HCI); Interaction devices; Sound-based input / output.

Robot Voices in Daily Life: Vocal Human-Likeness and Application Context as Determinants of User Acceptance

“…Unlike synthetic speech from earlier years, which often failed to produce convincing quality (e.g., Mayer et al, 2003;Atkinson et al, 2005), contemporary computer voices sound more and more natural (Craig and Schroeder, 2017). They prompt the idea that a phone call from a bot, for example, could soon be hardly distinguishable from a real person (Oord et al, 2016;Seaborn and Urakami, 2021)-unless a different design decision is made by the creators of the voice.…”

The Systematic Review-lution: A Manifesto to Promote Rigour and Inclusivity in Research Synthesis

“…Yet empirical work in HCI-viewed across the field as a whole-is disparate; how we report is varied and sometimes spotty. This is certainly not a novel criticism; we echo calls from other researchers who are critically reflecting on our research reporting practices, e.g., with regards to the reporting of race and ethnicity data [14], brain signal experiment data [68], participant compensation data [65], inter-rater reliability in qualitative research [57], specific measures [71] and questionnaires [43,47], engagement with self-determination theory [81], artifact descriptions [33], and inferential statistics [12], to list just a few. These issues may arise in part due to page limits or efforts to ensure paper length matches perceived contribution, but may also be due to lack of community-driven standardization and education.…”

Can Voice Assistants Sound Cute? Towards a Model of Kawaii Vocalics

“…Vocalics, or the paralinguistic qualities of voice, refer to "meta" vocal qualities beyond words, such pitch, volume, rate of speech, verbal fillers, and timbre, that convey information beyond speech content, notably emotion, personality, and social qualities, such as gender and age [38]. A variety of vocal qualities have been explored in computer voice over the past several decades [10,43,46]. Yet, kawaii voice is virtually unexplored.…”

“…For example, supplementary nonverbal kawaii sounds, like beeps and chortles, and sounds produced or recorded using bilabial mechanisms, for which Kumagai provides a starter list [20]. Exploratory work, such as qualitative perception studies, may elucidate this and seed future measures and instruments for kawaii vocalics, a crucial issue in voice UX work generally [46].…”