Uncertain Quality-Diversity: Evaluation Methodology and New Methods for Quality-Diversity in Uncertain Domains

Flageat, Manon; Cully, Antoine

doi:10.1109/tevc.2023.3273560

Cited by 3 publications

(2 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Reward Noise Another type of uncertainty considers additive noise directly on the reward value (Everitt et al 2017;Wang et al 2020;Romoff et al 2018), coming for example from imperfect sensors (Fig. 2.4).…”

Section: Uncertaintymentioning

confidence: 99%

See 1 more Smart Citation

Beyond Expected Return: Accounting for Policy Reproducibility When Evaluating Reinforcement Learning Algorithms

Flageat,

Lim,

Cully

2024

AAAI

View full text Add to dashboard Cite

Many applications in Reinforcement Learning (RL) usually have noise or stochasticity present in the environment. Beyond their impact on learning, these uncertainties lead the exact same policy to perform differently, i.e. yield different return, from one roll-out to another. Common evaluation procedures in RL summarise the consequent return distributions using solely the expected return, which does not account for the spread of the distribution. Our work defines this spread as the policy reproducibility: the ability of a policy to obtain similar performance when rolled out many times, a crucial property in some real-world applications. We highlight that existing procedures that only use the expected return are limited on two fronts: first an infinite number of return distributions with a wide range of performance-reproducibility trade-offs can have the same expected return, limiting its effectiveness when used for comparing policies; second, the expected return metric does not leave any room for practitioners to choose the best trade-off value for considered applications. In this work, we address these limitations by recommending the use of Lower Confidence Bound, a metric taken from Bayesian optimisation that provides the user with a preference parameter to choose a desired performance-reproducibility trade-off. We also formalise and quantify policy reproducibility, and demonstrate the benefit of our metrics using extensive experiments of popular RL algorithms on common uncertain RL tasks.

show abstract

Section: Uncertaintymentioning

confidence: 99%

“…Reward noise is a particular case of the reward corruption problem defined in Everitt et al (2017). However, while the noisy reward problem can be alleviated, Everitt et al (2017) derived a no-free lunch theorem for reward corruption.…”

Section: Uncertaintymentioning

confidence: 99%