Evaluation in artificial intelligence: from task-oriented to ability-oriented measurement

Abstract

The evaluation of artificial intelligence systems and components is crucial for the progress of the discipline. In this paper we describe and critically assess the different ways AI systems are evaluated, and the role of components and techniques in these systems. We first focus on the traditional task-oriented evaluation approach. We identify three kinds of evaluation: human discrimination, problem benchmarks and peer confrontation. We describe some of the limitations of the many evaluation schemes and competitions in these three categories, and follow the progression of some of these tests. We then focus on a less customary (and challenging) ability-oriented evaluation approach, where a system is characterised by its (cognitive) abilities, rather than by the tasks it is designed to solve. We discuss several possibilities: the adaptation of cognitive tests used for humans and animals, the development of tests derived from algorithmic information theory or more integrated approaches under the perspective of universal psychometrics. We analyse some evaluation tests from AI that are better positioned for an ability-oriented evaluation and discuss how their problems and limitations can possibly be addressed with some of the tools and ideas that appear within the paper. Finally, we enumerate a series of lessons learnt and generic guidelines to be used when an AI evaluation scheme is under consideration.

Innovation Potentials and Pathways Merging AI, CPS, and IoT

“…For a final outlook towards the impact of especially future AI developments [82][83][84][85][86][87][88][89] in the supply chain and business domain a metaphor might be adapted: Bostrom started his 2014 book about superintelligence-hence AI-with a fable of sparrows looking out for an owl, symbolizing humans on the verge of developing AI instances (see [14], p. v). However, this unfinished fable lacks three important features: First, sparrows and owls are-though quite different-still both quite similar bird species.…”

Computer Models Solving Intelligence Test Problems: Progress and Implications (Extended Abstract)

“…The categories and overlaps between problems could be assessed via theoretical models, instead of using factor analysis as in psychometrics. In other words, a theoretical alternative to the classification of mental abilities should be endeavoured (see Hernández-Orallo, 2016;Hernández-Orallo, 2017].…”

“…One way or the other, there seems to be an agreement that there will be an increasing number of machines in the near future which show a range of cognitive abilities, and that we will require evaluation mechanisms for them [Hernández-Orallo, 2017]. These mechanisms will have to give scores for several cognitive abilities so that we can compare them with humans and other animals.…”

“…The approach presented by Ohlsson et al [2013] is one of the few approaches to verbal intelligence tests. Hofmann, Kitzelmann, and Schmid [Hofmann et al, 2014] demonstrate that the inductive programming [Gulwani et al, 2015] system IGOR2 can be applied to number series problems. Martínez-Plumed et al [2017] introduced gErl, a declarative learning system able to solve several intelligence tests: Raven's Progressive Matrices, odd-one-out problems and letter series completion problems.…”

A New AI Evaluation Cosmos: Ready to Play the Game?

“…Actually, some of these platforms can integrate any task and hence in principle they supersede many existing AI benchmarks [2] in their aim to test "general problem solving ability".…”