Using fNIRS to Identify Transparency- and Reliability-Sensitive Markers of Trust Across Multiple Timescales in Collaborative Human-Human-Agent Triads

Abstract

Intelligent agents are rapidly evolving from assistants into teammates as they perform increasingly complex tasks. Successful human-agent teams leverage the computational power and sensory capabilities of automated agents while keeping the human operator's expectation consistent with the agent's ability. This helps prevent over-reliance on and under-utilization of the agent to optimize its effectiveness. Research at the intersection of human-computer interaction, social psychology, and neuroergonomics has identified trust as a governing factor of human-agent interactions that can be modulated to maintain an appropriate expectation. To achieve this calibration, trust can be monitored continuously and unobtrusively using neurophysiological sensors. While prior studies have demonstrated the potential of functional near-infrared spectroscopy (fNIRS), a lightweight neuroimaging technology, in the prediction of social, cognitive, and affective states, few have successfully used it to measure complex social constructs like trust in artificial agents. Even fewer studies have examined the dynamics of hybrid teams of more than 1 human or 1 agent. We address this gap by developing a highly collaborative task that requires knowledge sharing within teams of 2 humans and 1 agent. Using brain data obtained with fNIRS sensors, we aim to identify brain regions sensitive to changes in agent behavior on a long- and short-term scale. We manipulated agent reliability and transparency while measuring trust, mental demand, team processes, and affect. Transparency and reliability levels are found to significantly affect trust in the agent, while transparency explanations do not impact mental demand. Reducing agent communication is shown to disrupt interpersonal trust and team cohesion, suggesting similar dynamics as human-human teams. Contrasts of General Linear Model analyses identify dorsal medial prefrontal cortex activation specific to assessing the agent's transparency explanations and characterize increases in mental demand as signaled by dorsal lateral prefrontal cortex and frontopolar activation. Short scale event-level data is analyzed to show that predicting whether an individual will trust the agent, with data from 15 s before their decision, is feasible with fNIRS data. Discussing our results, we identify targets and directions for future neuroergonomics research as a step toward building an intelligent trust-modulation system to optimize human-agent collaborations in real time.

Stringesthesia: Dynamically Shifting Musical Agency Between Audience and Performer Based on Trust in an Interactive and Improvised Performance

“…As fNIRS is a newer neuroimaging technique than comparable technologies like fMRI and EEG, the literature using fNIRS to measure correlates of trust is limited [3,9,16]. However, since fNIRS and fMRI both measure the hemodynamics of the brain and have been proven to be correlated [28], we can draw from the extensive literature studying trust using fMRI (as summarized by [16]).…”

“…However, since fNIRS and fMRI both measure the hemodynamics of the brain and have been proven to be correlated [28], we can draw from the extensive literature studying trust using fMRI (as summarized by [16]). The main regions of interest (ROIs) implicated in decision-making and trust include the frontopolar area (FPA), medial prefrontal cortex (MPFC), dorsolateral prefrontal cortex (DLPFC), and the bilateral temporoparietal junction (TPJ) [9,16] (see Figure 2). The regions have often been implicated in Theory of Mind reasoning, a research paradigm evaluating how one attributes thoughts, intentions and beliefs to others [38].…”

“…The regions have often been implicated in Theory of Mind reasoning, a research paradigm evaluating how one attributes thoughts, intentions and beliefs to others [38]. These areas are often activated during social interactions, evaluation of self and others, decision-making, and determining when to trust [9,30,38]. Trust was further quantified using pilot data collected prior to the performance in which the performer played both cooperatively and competitively with another musician.…”

“…Trust is a crucial component of promoting performer-audience engagement, as both parties must express some level of confidence to carry out a successful performance [32]. While the interplay of trust has been explored in some interactive musical performances [31,32], much of the relevant trust literature comes from humanautomation interaction studies [3,9]. These studies evaluate trust often through the manipulation of an artificial agent's characteristics such as transparency and reliability.…”

“…In particular, when agents provide unreliable or incomplete information, their human collaborators are likely to ignore the information and rely solely on their own experience, resulting in a lack of trust in the agent. Conversely, when agents provide information that is overly reliable, humans may become complacent and over-rely on the agent, resulting in a lack of vigilance and degraded performance [9,13,17]. Communication or lack thereof between team members can reveal the level of trust, which can greatly impact the effectiveness of collaboration.…”