Advancing Human-AI Complementarity: The Impact of User Expertise and Algorithmic Tuning on Joint Decision Making

Abstract

Human-AI collaboration for decision-making strives to achieve team performance that exceeds the performance of humans or AI alone. However, many factors can impact success of Human-AI teams, including a user’s domain expertise, mental models of an AI system, trust in recommendations, and more. This paper reports on a study that examines users’ interactions with three simulated algorithmic models, all with equivalent accuracy rates but each tuned differently in terms of true positive and true negative rates. Our study examined user performance in a non-trivial blood vessel labeling task where participants indicated whether a given blood vessel was flowing or stalled. Users completed 150 trials across multiple stages, first without an AI and then with recommendations from an AI-Assistant. Although all users had prior experience with the task, their levels of proficiency varied widely. Our results demonstrated that while recommendations from an AI-Assistant can aid in users’ decision making, several underlying factors, including user base expertise and complementary human-AI tuning, significantly impact the overall team performance. First, users’ base performance matters, particularly in comparison to the performance level of the AI. Novice users improved, but not to the accuracy level of the AI. Highly proficient users were generally able to discern when they should follow the AI recommendation and typically maintained or improved their performance. Mid-performers, who had a similar level of accuracy to the AI, were most variable in terms of whether the AI recommendations helped or hurt their performance. Second, tuning an AI algorithm to complement users’ strengths and weaknesses also significantly impacted users’ performance. For example, users in our study were better at detecting flowing blood vessels, so when the AI was tuned to reduce false negatives (at the expense of increasing false positives), users were able to reject those recommendations more easily and improve in accuracy. Finally, users’ perception of the AI’s performance relative to their own performance had an impact on whether users’ accuracy improved when given recommendations from the AI. Overall, this work reveals important insights on the complex interplay of factors influencing Human-AI collaboration and provides recommendations on how to design and tune AI algorithms to complement users in decision-making tasks.

Interpretable AI in Medical Imaging: Enhancing Diagnostic Accuracy through Human-Computer Interaction

“…There is a need for further exploration into the interaction of these factors and their influence on collaborative decision-making between humans, which could offer valuable insights for refining decision-making processes across diverse contexts. A more comprehensive investigation in this direction would contribute to the improvement of decision-making strategies, accommodating individuals with diverse levels of expertise [30].…”

A Data-Intelligence-Intensive Bioinformatics Copilot System for Large-scale Omics Researches and Scientific Insights

“…It embeds knowledge from diverse scientific disciplines into AI agents, who, as integral members of intensive intelligence, actively participate in bioinformatics researches, facilitating interdisciplinary collaboration and overcoming long-standing knowledge barriers. Moreover, this novel research paradigm, leveraging the complementary integration of expert knowledge with AI capabilities 32,33 , accelerates bioinformatics workflows by feature extraction from big biological data, proposing innovative scientific hypotheses, optimizing research designs, orchestrating computational resources for high-performance computing, and instituting fair and objective evaluation metrics.…”