Comparing the Effectiveness of Speech and Physiological Features in Explaining Emotional Responses during Voice User Interface Interactions

Abstract

The rapid rise of voice user interface technology has changed the way users traditionally interact with interfaces, as tasks requiring gestural or visual attention are swapped by vocal commands. This shift has equally affected designers, required to disregard common digital interface guidelines in order to adapt to non-visual user interaction (No-UI) methods. The guidelines regarding voice user interface evaluation are far from the maturity of those surrounding digital interface evaluation, resulting in a lack of consensus and clarity. Thus, we sought to contribute to the emerging literature regarding voice user interface evaluation and, consequently, assist user experience professionals in their quest to create optimal vocal experiences. To do so, we compared the effectiveness of physiological features (e.g., phasic electrodermal activity amplitude) and speech features (e.g., spectral slope amplitude) to predict the intensity of users’ emotional responses during voice user interface interactions. We performed a within-subjects experiment in which the speech, facial expression, and electrodermal activity responses of 16 participants were recorded during voice user interface interactions that were purposely designed to elicit frustration and shock, resulting in 188 analyzed interactions. Our results suggest that the physiological measure of facial expression and its extracted feature, automatic facial expression-based valence, is most informative of emotional events lived through voice user interface interactions. By comparing the unique effectiveness of each feature, theoretical and practical contributions may be noted, as the results contribute to voice user interface literature while providing key insights favoring efficient voice user interface evaluation.

Feasibility of Electrodermal Activity and Photoplethysmography Data Acquisition at the Foot Using a Sock Form Factor

“…This sensor is placed at the ankle, as this location allows a suitable integration of the sensor in the sock and has been reported as a suitable anatomical location for PPG acquisitions [ 10 ]. For the EDA acquisition, the BITalino EDA 081217 sensor (from PLUX, S.A., Lisbon, Portugal) was considered; this sensor works under the DC exosomatic measurement method, has been used in multiple research projects [ 37 , 38 , 39 ] and has a conductance measurement range of S.…”

“…As a reference device, the BITalino (r)evolution Plugged (PLUX, S.A., Lisbon, Portugal) was used, which was connected to the same type of sensors. It is worth mentioning that the BITalino has been widely used within the research community for EDA acquisitions using the BITalino EDA sensor (as used in this experiment) [ 37 , 38 , 39 , 41 ]. Furthermore, this device has also been used for PPG acquisitions in laboratory research contexts [ 42 , 43 ].…”

Waiting Experience: Optimization of Feedback Mechanism of Voice User Interfaces Based on Time Perception

“…In the process of user voice interaction, the user's emotion could be affected by the response of VUI, which obtained some researcher's interest. Swoboda et al [10] compared the effectiveness of physiological features and speech features to predict the intensity of users' emotional responses during voice user interface interactions. Their research results suggest that the physiological measure of facial expression and its extracted feature, automatic facial expression-based valence, is the most informative of emotional events lived through voice user interface interactions.…”

Speech capacity in patients with voice disorders before and after surgical treatment of vocal fold tumors

“…Voice is a result of laryngeal sound production which is primarily an auditory component of speech that allows the expression of emotions and feelings in personal, social and professional relationships [1,2]. As part of the speech capacity, voice can be quantified by using acoustic voice parameters, including fundamental frequency, intensity, quality, melody [3] and so on. Thus, voice is associated with prosody [4], emotions [3,5,6], temporal organization of speech [7][8][9], articulation of sounds, phonemes, and melody of speech [10].…”

“…As part of the speech capacity, voice can be quantified by using acoustic voice parameters, including fundamental frequency, intensity, quality, melody [3] and so on. Thus, voice is associated with prosody [4], emotions [3,5,6], temporal organization of speech [7][8][9], articulation of sounds, phonemes, and melody of speech [10].…”