On environment difficulty and discriminating power

Hernández-Orallo, José

doi:10.1007/s10458-014-9257-1

Cited by 12 publications

(16 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We set tasks and agents as asynchronous interactive systems, where difficulty is seen as computational steps of a Levin search, but this search has to be modified to cover stochastic behaviours. These ideas are an evolution and continuation of early notions of task and difficulty in [8] and [6] respectively. The relevance of verification in difficulty has usually been associated with deduction.…”

Section: Difficulty As Levin Search With Stochastic Verificationmentioning

confidence: 93%

Stochastic Tasks: Difficulty and Levin Search

Hernández-Orallo

2015

Artificial General Intelligence

Self Cite

View full text Add to dashboard Cite

Abstract. We establish a setting for asynchronous stochastic tasks that account for episodes, rewards and responses, and, most especially, the computational complexity of the algorithm behind an agent solving a task. This is used to determine the difficulty of a task as the (logarithm of the) number of computational steps required to acquire an acceptable policy for the task, which includes the exploration of policies and their verification. We also analyse instance difficulty, task compositions and decompositions.

show abstract

Section: Difficulty As Levin Search With Stochastic Verificationmentioning

confidence: 93%

Stochastic Tasks: Difficulty and Levin Search

Hernández-Orallo

2015

Artificial General Intelligence

Self Cite

View full text Add to dashboard Cite

show abstract

“…[43]), one simple notion that accounts for this concept quite well is the variance of results. In order to formalise this notion of variance (or number of different values) of the expected result of the set of evaluated agents, we can just compare pairs of values as follows:…”

Section: Fine and Coarse Discriminationmentioning

confidence: 99%

“…Every test should provide which level of difficulty it is evaluating. The difficulty of the environment could be calculated as, for example, the performance of a distribution of agents' policies with different levels of complexity, as presented in [43]. We postulated that the level of difficulty should be determined by the agents included in the environment (and their intelligence), the partition of agent slots that determines how teams are formed and the environment where the agent is evaluated.…”

Section: Our Definition Uses Various Sets and Weights In The Formula mentioning

confidence: 99%

Towards a Universal Test of Social Intelligence

Cabrera¹

View full text Add to dashboard Cite

Under the view of artificial intelligence, an intelligent agent is an autonomous entity which interacts in an environment through observations and actions, trying to achieve one or more goals with the aid of several signals called rewards. The creation of intelligent agents is proliferating during the last decades, and the evaluation of their intelligence is a fundamental issue for their understanding, construction and improvement.Social intelligence is recently obtaining special attention in the creation of intelligent agents due to the current view of human intelligence as highly social. Social intelligence in natural and artificial systems is usually measured by the evaluation of associated traits or tasks that are deemed to represent some facets of social behaviour. The amalgamation of these traits or tasks is then used to configure an operative notion of social intelligence. However, this operative notion does not truly represent what social intelligence is and a definition following this principle will not be precise. Instead, in this thesis we investigate the evaluation of social intelligence in a more formal and general way, by actually considering the evaluee's interaction with other agents.In this thesis we analyse the implications of evaluating social intelligence using a test that evaluates general intelligence. For this purpose, we include other agents into an initially singleagent environment to figure out the issues that appear when evaluating an agent in the context of other agents. From this analysis we obtain useful information for the evaluation of social intelligence.From the lessons learned, we identify the components that should be considered in order to measure social intelligence, and we provide a formal and parametrised definition of social intelligence. This definition calculates an agent's social intelligence as its expected performance in a set of environments with a set of other agents arranged in teams and participating in lineups, with rewards being re-understood appropriately. This is conceived as a tool to define social intelligence testbeds where we can generate several degrees of competitive and cooperative behaviours. We test this definition by experimentally analysing the influence of teams and agent line-ups for several multi-agent systems with variants of Q-learning agents.However, not all testbeds are appropriate for the evaluation of social intelligence. To facilitate the analysis of a social intelligence testbed, we provide some formal property models about social intelligence in order to characterise the testbed and thus assess its suitability. Finally, we use the presented properties to characterise some social games and multi-agent environments, we make a comparison between them and discuss their strengths and weaknesses in order to evaluate social intelligence.ii ResumenBajo la visión de la inteligencia artificial, un agente inteligente es una entidad autónoma la cual interactúa en un entorno a través de observaciones y acciones, tratando de lograr uno o más ...

show abstract

“…In fact, a league may be redundant (for the same reasons why the information-driven or difficultydriven sampling are introduced) and other tournament arrangements are more effective with almost the same robustness and much fewer matches. q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q 100 104 185 161 167 180 224 194 174 155 118 106 58 40 23 8 2 1 100 1 2 5 16 39 79 109 123 135 112 101 57 40 23 8 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q 100 106 193 171 172 178 198 187 195 158 120 89 70 39 16 5 3 100 1 2 5 13 39 72 114 152 143 113 87 68 39 16 5 3 Figure 4: We show the distributions of reward (roughly corresponding to R in this paper) for different configurations for the multi-agent system SCMAS introduced in [70]. Left: the plot shows the results when we confront each of the 2,000 policies with 50 different teams of competitors (with different seeds for the generator also).…”

Section: Evaluation By Peer Confrontationmentioning

confidence: 99%

Evaluation in artificial intelligence: from task-oriented to ability-oriented measurement

Hernández-Orallo

2016

Artif Intell Rev

Self Cite

108

View full text Add to dashboard Cite

The evaluation of artificial intelligence systems and components is crucial for the progress of the discipline. In this paper we describe and critically assess the different ways AI systems are evaluated, and the role of components and techniques in these systems. We first focus on the traditional task-oriented evaluation approach. We identify three kinds of evaluation: human discrimination, problem benchmarks and peer confrontation. We describe some of the limitations of the many evaluation schemes and competitions in these three categories, and follow the progression of some of these tests. We then focus on a less customary (and challenging) ability-oriented evaluation approach, where a system is characterised by its (cognitive) abilities, rather than by the tasks it is designed to solve. We discuss several possibilities: the adaptation of cognitive tests used for humans and animals, the development of tests derived from algorithmic information theory or more integrated approaches under the perspective of universal psychometrics. We analyse some evaluation tests from AI that are better positioned for an ability-oriented evaluation and discuss how their problems and limitations can possibly be addressed with some of the tools and ideas that appear within the paper. Finally, we enumerate a series of lessons learnt and generic guidelines to be used when an AI evaluation scheme is under consideration.

show abstract

On environment difficulty and discriminating power

Cited by 12 publications

References 45 publications

Stochastic Tasks: Difficulty and Levin Search

Stochastic Tasks: Difficulty and Levin Search

Towards a Universal Test of Social Intelligence

Evaluation in artificial intelligence: from task-oriented to ability-oriented measurement

Contact Info

Product

Resources

About