Human and Machine: The Impact of Machine Input on Decision-Making Under Cognitive Limitations

Boyacı, Tamer; Canyakmaz, Caner; deVericourt, Francis

doi:10.2139/ssrn.3740508

Cited by 5 publications

(6 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Embracing the vision of multiplying diverse expertise, many scholars describe human-AI augmentation as humans and machines "collaborating" together (e.g., Boyaci et al 2020, Gao et al 2021, Khadpe et al 2020, Puranam 2021. Prior organizational literature on effective collaboration among diverse human experts shows how experts learn ways of working together to leverage and combine the complementary capabilities (Hardy et al 2005, Maguire et al 2004.…”

Section: Augmenting Professional Expertise With Aimentioning

confidence: 99%

To Engage or Not to Engage with AI for Critical Judgments: How Professionals Deal with Opacity When Using AI for Medical Diagnosis

Lebovitz

Lifshitz-Assaf

Levina

2022

Organization Science

185

View full text Add to dashboard Cite

Artificial intelligence (AI) technologies promise to transform how professionals conduct knowledge work by augmenting their capabilities for making professional judgments. We know little, however, about how human-AI augmentation takes place in practice. Yet, gaining this understanding is particularly important when professionals use AI tools to form judgments on critical decisions. We conducted an in-depth field study in a major U.S. hospital where AI tools were used in three departments by diagnostic radiologists making breast cancer, lung cancer, and bone age determinations. The study illustrates the hindering effects of opacity that professionals experienced when using AI tools and explores how these professionals grappled with it in practice. In all three departments, this opacity resulted in professionals experiencing increased uncertainty because AI tool results often diverged from their initial judgment without providing underlying reasoning. Only in one department (of the three) did professionals consistently incorporate AI results into their final judgments, achieving what we call engaged augmentation. These professionals invested in AI interrogation practices—practices enacted by human experts to relate their own knowledge claims to AI knowledge claims. Professionals in the other two departments did not enact such practices and did not incorporate AI inputs into their final decisions, which we call unengaged “augmentation.” Our study unpacks the challenges involved in augmenting professional judgment with powerful, yet opaque, technologies and contributes to literature on AI adoption in knowledge work.

show abstract

Section: Augmenting Professional Expertise With Aimentioning

confidence: 99%

To Engage or Not to Engage with AI for Critical Judgments: How Professionals Deal with Opacity When Using AI for Medical Diagnosis

Lebovitz

Lifshitz-Assaf

Levina

2022

Organization Science

185

View full text Add to dashboard Cite

show abstract

“…High-AIQ individuals excel at maximizing the unique strengths of both humans and AI while minimizing their respective weaknesses. As an individual difference, AIQ captures the notion that individuals vary in the ability to recognize human-AI complementarities, delegate tasks to AI, incorporate AI's output, refine these processes, and so forth (20). This is evidenced by anecdotes from diverse contexts.…”

Section: Artificial Intelligence Quotient (Aiq) "The Power Of Working...mentioning

confidence: 99%

Artificial Intelligence Quotient (AIQ)

Qin,

Lu,

Chen

et al. 2024

Preprint

View full text Add to dashboard Cite

We introduce the concept of Artificial Intelligence Quotient (AIQ)—operationally defined as a person’s ability to use AI to perform a wide variety of tasks—and provide evidence for its existence across archival and lab studies. Study 1 (a 19-year global dataset of human+AI chess tournaments) and Study 2 (a three-wave longitudinal study of human+AI renju games) show that individuals have stable human+AI performance over time (controlling for human’s own capability and AI’s capability), suggesting the existence of a stable human+AI capability. Study 3 shows that a general AIQ factor can be statistically extracted from individuals’ performance on a variety of tasks completed with ChatGPT, a more general and widely-used AI tool. Besides replicating Study 3’s findings in a larger sample, Study 4 further shows that the extracted AIQ factor predicts human+AI performance not only on a new task using the same AI (ChatGPT), but also on another new task using a different AI (renju AI) a week later. Across these studies, we ascertain the unique explanatory power of AIQ by controlling for individual’s computer ability, intelligence quotient (IQ), and/or emotional intelligence (EQ). We also explored potential correlates of AIQ (personality traits, perceived AI agency and feelings, previous AI use, and demographics). Together, our findings suggest that AIQ exists and is measurable. By establishing this new type of intelligence (AIQ), we shed light on individual differences in the ability to use AI, which is essential for individuals, organizations, and society in an era of AI.

show abstract

“…This approach is commonly used to model mistrust bias ( Özer and Zheng 2018), in particular when a human makes decisions based on the input of a data-driven algorithm (Ahsen et al 2019, Boyaci et al 2020. Specifically, the DM's updated belief that an action is required given the DM's and machine's signals is a linear combination between the updated belief of the human and that of the machine.…”

Section: Mistrusting the Machine When Decidingmentioning

confidence: 99%

Is Your Machine Better Than You? You May Never Know

Véricourt

Gürkan

2023

Management Science

View full text Add to dashboard Cite

Artificial intelligence systems are increasingly demonstrating their capacity to make better predictions than human experts. Yet recent studies suggest that professionals sometimes doubt the quality of these systems and overrule machine-based prescriptions. This paper explores the extent to which a decision maker (DM) supervising a machine to make high-stakes decisions can properly assess whether the machine produces better recommendations. To that end, we study a setup in which a machine performs repeated decision tasks (e.g., whether to perform a biopsy) under the DM’s supervision. Because stakes are high, the DM primarily focuses on making the best choice for the task at hand. Nonetheless, as the DM observes the correctness of the machine’s prescriptions across tasks, the DM updates the DM’s belief about the machine. However, the DM is subject to a so-called verification bias such that the DM verifies the machine’s correctness and updates the DM’s belief accordingly only if the DM ultimately decides to act on the task. In this setup, we characterize the evolution of the DM’s belief and overruling decisions over time. We identify situations under which the DM hesitates forever whether the machine is better; that is, the DM never fully ignores but regularly overrules it. Moreover, the DM sometimes wrongly believes with positive probability that the machine is better. We fully characterize the conditions under which these learning failures occur and explore how mistrusting the machine affects them. These findings provide a novel explanation for human–machine complementarity and suggest guidelines on the decision to fully adopt or reject a machine. This paper was accepted by Elena Katok, special issue on the human–algorithm connection. Supplemental Material: The online appendix is available at https://doi.org/10.1287/mnsc.2023.4791 .

show abstract

Human and Machine: The Impact of Machine Input on Decision-Making Under Cognitive Limitations

Abstract: All rights reserved. This document may be distributed for free -electronically or in print -in the same formats as it is available on the website of the ESMT (www.esmt.org) for non-commercial purposes. It is not allowed to produce any derivates of it without the written permission of ESMT.

Cited by 5 publications

References 52 publications

To Engage or Not to Engage with AI for Critical Judgments: How Professionals Deal with Opacity When Using AI for Medical Diagnosis

To Engage or Not to Engage with AI for Critical Judgments: How Professionals Deal with Opacity When Using AI for Medical Diagnosis

Artificial Intelligence Quotient (AIQ)

Is Your Machine Better Than You? You May Never Know

Contact Info

Product

Resources

About