Paradoxes in Numerical Comparison of Optimization Algorithms

Abstract

Numerical comparison is often key to verifying the performance of optimization algorithms, especially, global optimization algorithms. However, studies have so far neglected issues concerning comparison strategies necessary to rank optimization algorithms properly. To fill this gap for the first time, we combine voting theory and numerical comparison research areas, which have been disjoint so far, and thus extend the results of the former to the latter for optimization algorithms. In particular, we investigate compatibility issues arising from comparing two and more than two algorithms, termed "C2" and "C2+" in this article, respectively. Through defining and modeling "C2" and "C2+" mathematically, it is uncovered and illustrated that numerical comparison can be incompatible. Further, two possible paradoxes, namely, "cycle ranking" and "survival of the nonfittest," are discovered and analyzed rigorously. The occurrence probabilities of these two paradoxes are also calculated under the no-free-lunch assumption, which shows the first justifiable use of the impartial culture assumption from voting theory, providing a point of reference to the frequency of the paradoxes occurring. It is also shown that significant influence on these probabilities comes from the number of algorithms and the number of optimization problems studied in the comparison. Further, various limiting probabilities when the number of optimization problems goes to infinity are also derived and characterized. The results would help guide benchmarking and developing optimization and machine learning algorithms.

Metaheuristic Optimization of Power and Energy Systems: Underlying Principles and Main Issues of the ‘Rush to Heuristics’

“…The quality of the algorithms can then be determined by calculating the percentage of the best solutions, averaged over a given number of executions of each algorithm [13]. The series of the best solutions obtained during the execution of the algorithm in the given time is typically considered for applying a performance metric [29,92].…”

“…However, a systematic guide on how to select the set of problems is still missing. The hint given in Liu et al (2020) is to select the whole set of optimization problems in a given domain, and not only a partial set.…”

“…Indications on the comparison strategies are provided in Liu et al (2020), basically identifying pairwise comparison between algorithms (with the variants one-plays-all, generally used to check a new algorithm, and all-play-all or "round-robin"), and multi-algorithm comparison, both used in many contexts. For multi-algorithm comparison, statistical aggregations such as the cumulative distribution function are often used.…”

“…The latter type of comparison has become more and more interesting, also representing the confidence intervals (Liu et al 2017;Doerr et al 2018). Liu et al (2020) defined the problem of finding the best algorithm as a voting system, in which the algorithms are the candidates, the problems are the votes, and an algorithm performs better than the others if it exhibits better performance on more problems. However, they found the existence of the so-called "cycle ranking" or Condorcet paradox, namely, it may happen that different algorithms are winners for different problems, and it is not possible to conclude which algorithm is better overall.…”

“…A recent contribution (Liu et al 2020) has addressed comparison strategies and their mathematical properties in a systematic way. The numerical comparison between optimization algorithms consists of the selection of a set of algorithms and problems, the testing of the algorithms on the problems, the identification of comparison strategy, methods and metrics, the analysis of the outcomes obtained from applying the metrics, and the final determination of the results.…”