Interaction of Thoughts: Towards Mediating Task Assignment in Human-AI Cooperation with a Capability-Aware Shared Mental Model

He, Ziyao; Song, Yunpeng; Zhou, Shurui; Cai, Zhongmin

doi:10.1145/3544548.3580983

Cited by 7 publications

(1 citation statement)

References 53 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…By computing Hedges' g, a unitless standardized effect size, we can describe important relations among these experiments in ways that make them comparable across different study designs with different outcome variables [36]. The studies in our data set, though, come from different samples of people -some look at doctors [37][38][39], others at crowdworkers [4,6,34], and still others at students [13,40,41] -and this variation can limit the comparability of the effect size to a degree [36]. Additionally, the measurement error can also vary across experiments.…”

Section: Surprisingly Insignificant Moderatorsmentioning

confidence: 99%

When Are Combinations of Humans and AI Useful?

Vaccaro,

Almaatouq,

Malone

2024

Preprint

View full text Add to dashboard Cite

Inspired by the increasing use of AI to augment humans, researchers have studied human-AI systems involving different tasks, systems, and populations. Despite such a large body of work, we lack a broad conceptual understanding of when combinations of humans and AI are better than either alone. Here, we addressed this question by conducting a meta-analysis of over 100 recent experimental studies reporting over 300 effect sizes. First, we found that, on average, human-AI combinations performed significantly worse than the best of humans or AI alone. Second, we found performance losses in tasks that involved making decisions and significantly greater gains in tasks that involved creating content. Finally, when humans outperformed AI alone, we found performance gains in the combination, but when the AI outperformed humans alone we found losses. These findings highlight the heterogeneity of the effects of human-AI collaboration and point to promising avenues for improving human-AI systems.

show abstract

Section: Surprisingly Insignificant Moderatorsmentioning

confidence: 99%

When Are Combinations of Humans and AI Useful?

Vaccaro,

Almaatouq,

Malone

2024

Preprint

View full text Add to dashboard Cite

show abstract

When combinations of humans and AI are useful: A systematic review and meta-analysis

Vaccaro,

Almaatouq,

Malone

2024

Nat Hum Behav

View full text Add to dashboard Cite

Inspired by the increasing use of artificial intelligence (AI) to augment humans, researchers have studied human–AI systems involving different tasks, systems and populations. Despite such a large body of work, we lack a broad conceptual understanding of when combinations of humans and AI are better than either alone. Here we addressed this question by conducting a preregistered systematic review and meta-analysis of 106 experimental studies reporting 370 effect sizes. We searched an interdisciplinary set of databases (the Association for Computing Machinery Digital Library, the Web of Science and the Association for Information Systems eLibrary) for studies published between 1 January 2020 and 30 June 2023. Each study was required to include an original human-participants experiment that evaluated the performance of humans alone, AI alone and human–AI combinations. First, we found that, on average, human–AI combinations performed significantly worse than the best of humans or AI alone (Hedges’ g = −0.23; 95% confidence interval, −0.39 to −0.07). Second, we found performance losses in tasks that involved making decisions and significantly greater gains in tasks that involved creating content. Finally, when humans outperformed AI alone, we found performance gains in the combination, but when AI outperformed humans alone, we found losses. Limitations of the evidence assessed here include possible publication bias and variations in the study designs analysed. Overall, these findings highlight the heterogeneity of the effects of human–AI collaboration and point to promising avenues for improving human–AI systems.

show abstract