Learning to Work in a Materials Recovery Facility: Can Humans and Machines Learn from Each Other?

Kyriacou, Harrison; Ramakrishnan, Anand; Whitehill, Jacob

doi:10.1145/3448139.3448183

Cited by 2 publications

(1 citation statement)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most of the ratios less than 1.0 come from experiments in which the human-computer systems perform worse than the computer alone according to some measure of accuracy, sensitivity, or specificity [38,39,40]. At the bottom of the distribution, for example, [40] conduct an experiment in which participants view an image of a meal and must substitute the highest carbohydrate ingredient with a low carbohydrate ingredient still similar in flavor.…”

Section: Study 1: Analysis Of Recent Studies That Evaluate Human-comp...mentioning

confidence: 99%

A Test for Evaluating Performance in Human-AI Systems

Malone

Vaccaro

Campero

et al. 2023

Preprint

View full text Add to dashboard Cite

Many important uses of AI involve augmenting humans, not replacing them. But there is not yet a widely used and broadly comparable test for evaluating the performance of these human-AI systems relative to humans alone, AI alone, or other baselines. Here we describe such a test and demonstrate its use in three ways. First, in an analysis of 79 recently published results, we find that, surprisingly, the median performance improvement ratio corresponds to no improvement at all, and the maximum improvement is only 36%. Second, we experimentally find a 27% performance improvement when 100 human programmers develop software using GPT-3, a modern, generative AI system. Finally, we find that 50 human non-programmers using GPT-3 perform the task about as well as –- and less expensively than –- the human programmers. Since neither the non-programmers nor the computer could perform the task alone, this illustrates a strong form of human-AI synergy.

show abstract