Trust Engineering for Human-AI Teams

Ezer, Neta; Bruni, Sylvain; Cai, Yang; Hepenstal, Sam; Miller, Christopher A.; Schmorrow, Dylan D.

doi:10.1177/1071181319631264

Cited by 46 publications

(20 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the one hand, the AI system needs to be trained with appropriate data sets to increase accuracy. On the other hand, humans need to learn to understand the DPA's co-regulation terminologies, concepts, and processes (Ezer et al 2019), which emphasises the role of explanation in a broad sense.…”

Section: Discussionmentioning

confidence: 99%

Partnering with AI: the case of digital productivity assistants

Cranefield

Winikoff

Chiu

et al. 2022

Journal of the Royal Society of New Zealand

View full text Add to dashboard Cite

An emerging class of intelligent tools that we term Digital Productivity Assistants (DPAs) is designed to help workers improve their productivity and keep their work-life balance in check. Using personalised work-based analytics it raises awareness of individual collaboration behaviour and suggests improvements to work practices. The purpose of this study is to contribute to a better understanding of the role of personalised work-based analytics in the context of (improving) individual productivity and work-life balance. We present an interpretive case study based on interviews with 28 workers who face high job demands and job variety and our own observations. Our study contributes to the still ongoing sensemaking of AI, by illustrating how DPAs can co-regulate human work through technology affordances. In addition to investigating these opportunities of partnering with AI, we study the perceived barriers that impede DPAs' potential benefits as partners. These include perceived accuracy, transparency, feedback, and configurability, as well as misalignment between the DPA's categorisations of work behaviour and the categorisations used by workers in their jobs.

show abstract

Section: Discussionmentioning

confidence: 99%

Partnering with AI: the case of digital productivity assistants

Cranefield

Winikoff

Chiu

et al. 2022

Journal of the Royal Society of New Zealand

View full text Add to dashboard Cite

show abstract

“…To achieve effective teaming between an analyst and a CA, the analyst must be able to trust the information retrieved by the CA and the processing involved. A key aspect enabling trust engineering in systems and addressing ethical concerns is transparency, so that the analyst can predict, interpret and refute any results, acknowledging caveats where they exist [11]. While trust is a crucial element in enabling analysts to team with AI systems, it needs to be handled carefully.…”

Section: Algorithmic Transparency Of Conversational Agentsmentioning

confidence: 99%

Developing Conversational Agents for Use in Criminal Investigations

Hepenstal

Zhang

Kodagoda

et al. 2021

ACM Trans. Interact. Intell. Syst.

View full text Add to dashboard Cite

The adoption of artificial intelligence (AI) systems in environments that involve high risk and high consequence decision-making is severely hampered by critical design issues. These issues include system transparency and brittleness, where transparency relates to (i) the explainability of results and (ii) the ability of a user to inspect and verify system goals and constraints; and brittleness, (iii) the ability of a system to adapt to new user demands. Transparency is a particular concern for criminal intelligence analysis, where there are significant ethical and trust issues that arise when algorithmic and system processes are not adequately understood by a user. This prevents adoption of potentially useful technologies in policing environments. In this article, we present a novel approach to designing a conversational agent (CA) AI system for intelligence analysis that tackles these issues. We discuss the results and implications of three different studies; a Cognitive Task Analysis to understand analyst thinking when retrieving information in an investigation, Emergent Themes Analysis to understand the explanation needs of different system components, and an interactive experiment with a prototype conversational agent. Our prototype conversational agent, named Pan, demonstrates transparency provision and mitigates brittleness by evolving new CA intentions. We encode interactions with the CA with human factors principles for situation recognition and use interactive visual analytics to support analyst reasoning. Our approach enables complex AI systems, such as Pan, to be used in sensitive environments, and our research has broader application than the use case discussed.

show abstract

“…The recent proposal of Trust Engineering for human-AI teaming by Ezer et al [60] insisted that there are still many challenges in managing trust in AI systems that are increasingly complex and work within imperfect information environments. They proposed six conceptual components in Trust Engineering: adaptability, communication, explainability, training/knowledge, assessment, and security.…”

Section: Discussionmentioning

confidence: 99%