It’s Not Only What You Say, But Also How You Say It: Machine Learning Approach to Estimate Trust from Conversation

Abstract

Objective The objective of this study was to estimate trust from conversations using both lexical and acoustic data. Background As NASA moves to long-duration space exploration operations, the increasing need for cooperation between humans and virtual agents requires real-time trust estimation by virtual agents. Measuring trust through conversation is a novel and unintrusive approach. Method A 2 (reliability) × 2 (cycles) × 3 (events) within-subject study with habitat system maintenance was designed to elicit various levels of trust in a conversational agent. Participants had trust-related conversations with the conversational agent at the end of each decision-making task. To estimate trust, subjective trust ratings were predicted using machine learning models trained on three types of conversational features (i.e., lexical, acoustic, and combined). After training, model explanation was performed using variable importance and partial dependence plots. Results Results showed that a random forest algorithm, trained using the combined lexical and acoustic features, predicted trust in the conversational agent most accurately [Formula: see text]. The most important predictors were a combination of lexical and acoustic cues: average sentiment considering valence shifters, the mean of formants, and Mel-frequency cepstral coefficients (MFCC). These conversational features were identified as partial mediators predicting people’s trust. Conclusion Precise trust estimation from conversation requires lexical cues and acoustic cues. Application These results showed the possibility of using conversational data to measure trust, and potentially other dynamic mental states, unobtrusively and dynamically.

Collaborative System Usability in Spaceflight Analog Environments through Remote Observations

“…As a proposed cornerstone of this framework, voice recordings in particular [9][10][11] have garnered much recent attention as promising enhancements to usability evaluation. Work by Li and colleagues [12] suggests that trust in autonomous agents can be inferred from lexical and acoustic features in speech. Similarly, Magnusdottir et al [13] coupled cardiovascular measures and speech to measure the dynamics of cognitive workload during various tasks (see also [14]).…”