Quantitative Performance Assessment of Multiobjective Optimizers: The Average Runtime Attainment Function

Brockhoff, Dimo; Auger, Anne; Hansen, Nikolaus; Tušar, Tea

doi:10.1007/978-3-319-54157-0_8

Cited by 11 publications

(6 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…7 As a counterexample, minimizing the third decimal digit of the quality indicator would be a well defined and achievable goal but neither relevant nor meaningful in practice, unless also the preceding digits are minimal. 8 For example, minimizing the norm of the gradient is meaningful and relevant on unimodal differentiable functions, but the gradient is often not available in the blackbox optimization setting.…”

Section: A Quality Indicatorsmentioning

confidence: 99%

Anytime Performance Assessment in Blackbox Optimization Benchmarking

Hansen

Auger

Brockhoff

et al. 2022

IEEE Trans. Evol. Computat.

Self Cite

View full text Add to dashboard Cite

We present concepts and recipes for the anytime performance assessment when benchmarking optimization algorithms in a blackbox scenario. We consider runtime-oftentimes measured in number of blackbox evaluations needed to reach a target quality-to be a universally measurable cost for solving a problem. Starting from the graph that depicts the solution quality versus runtime, we argue that runtime is the only performance measure with a generic, meaningful, and quantitative interpretation. Hence, our assessment is solely based on runtime measurements. We discuss proper choices for solution quality indicators in single-and multiobjective optimization, as well as in the presence of noise and constraints. We also discuss the choice of the target values, budget-based targets, and the aggregation of runtimes by using simulated restarts, averages, and empirical cumulative distributions which generalize convergence graphs of single runs. The presented performance assessment is to a large extent implemented in the COCO (Comparing Continuous Optimizers) platform freely available at https://github.com/numbbo/coco.

show abstract

Section: A Quality Indicatorsmentioning

confidence: 99%

Anytime Performance Assessment in Blackbox Optimization Benchmarking

Hansen

Auger

Brockhoff

et al. 2022

IEEE Trans. Evol. Computat.

Self Cite

View full text Add to dashboard Cite

show abstract

“…An alternative to the EAF method, Brockhoff et al (2017) proposed the average runtime attainment function, aRTA to measure the expected runtime of solution that weakly dominates PF. This method is a generalization of attainment function that based on a target vector z ∈ ℝ d to collect the runtime,T(z) as the minimum number of function evaluations to obtain the solution that weakly dominates z.…”

Section: Average Runtime Attainment Function Arta (Brockhoff Et Al 2017)mentioning

confidence: 99%

“…(31). The aRTA function maps ℝ d to positive real numbers,ℝ + with a color map as demonstrated in Brockhoff et al (2017). The advantage of this method compared to EAF is that aRTA can capture the algorithm's performance over multiple runs and over time, whereas EAF captures only on n defined times.…”

Section: Average Runtime Attainment Function Arta (Brockhoff Et Al 2017)mentioning

confidence: 99%

Performance assessment of the metaheuristic optimization algorithms: an exhaustive review

2020

View full text Add to dashboard Cite

The simulation-driven metaheuristic algorithms have been successful in solving numerous problems compared to their deterministic counterparts. Despite this advantage, the stochastic nature of such algorithms resulted in a spectrum of solutions by a certain number of trials that may lead to the uncertainty of quality solutions. Therefore, it is of utmost importance to use a correct tool for measuring the performance of the diverse set of metaheuristic algorithms to derive an appropriate judgment on the superiority of the algorithms and also to validate the claims raised by researchers for their specific objectives. The performance of a randomized metaheuristic algorithm can be divided into efficiency and effectiveness measures. The efficiency relates to the algorithm's speed of finding accurate solutions, convergence, and computation. On the other hand, effectiveness relates to the algorithm's capability of finding quality solutions. Both scopes are crucial for continuous and discrete problems either in single-or multi-objectives. Each problem type has different formulation and methods of measurement within the scope of efficiency and effectiveness performance. One of the most decisive verdicts for the effectiveness measure is the statistical analysis that depends on the data distribution and appropriate tool for correct judgments.

show abstract

“…Other examples of visual methods for examining algorithm parameters are [2,11]. Existing work from [1] visualises search history in EMO.…”

Section: Previous Visualising Search History Literaturementioning

confidence: 99%

Visualising Evolution History in Multi- and Many-Objective Optimisation

Walter¹,

Walker²,

Craven³

2020

Preprint

View full text Add to dashboard Cite

Evolutionary algorithms are widely used to solve optimisation problems. However, challenges of transparency arise in both visualising the processes of an optimiser operating through a problem and understanding the problem features produced from many-objective problems, where comprehending four or more spatial dimensions is difficult. This work considers the visualisation of a population as an optimisation process executes. We have adapted an existing visualisation technique to multi-and many-objective problem data, enabling a user to visualise the EA processes and identify specific problem characteristics and thus providing a greater understanding of the problem landscape. This is particularly valuable if the problem landscape is unknown, contains unknown features or is a many-objective problem. We have shown how using this framework is effective on a suite of multi-and many-objective benchmark test problems, optimising them with NSGA-II and NSGA-III.

show abstract

Quantitative Performance Assessment of Multiobjective Optimizers: The Average Runtime Attainment Function

Cited by 11 publications

References 12 publications

Anytime Performance Assessment in Blackbox Optimization Benchmarking

Anytime Performance Assessment in Blackbox Optimization Benchmarking

Performance assessment of the metaheuristic optimization algorithms: an exhaustive review

Visualising Evolution History in Multi- and Many-Objective Optimisation

Contact Info

Product

Resources

About