Surrogate-Assisted Hybrid-Model Estimation of Distribution Algorithm for Mixed-Variable Hyperparameters Optimization in Convolutional Neural Networks

Li, Jianyu; Zhan, Zhi-Hui; Xu, Jinan; Zhang, Mengjie

doi:10.1109/tnnls.2021.3106399

Cited by 39 publications

(25 citation statements)

References 62 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For future work, the proposed algorithm will be further extended to solving more difficult and complex MTOPs, such as not only in complex continuous space [54]- [56], but also in complex discrete [57]- [60], combinational [61]- [64], and mix-variable space [65]- [67]. Furthermore, as the MKT is a generic idea, further exploration of other kinds of meta-knowledge and other meta-knowledge transfer methods and utilization methods are worthy studied to obtain more powerful EMTO algorithms.…”

Section: Discussionmentioning

confidence: 99%

A Meta-Knowledge Transfer-Based Differential Evolution for Multitask Optimization

Zhan

Tan

et al. 2022

IEEE Trans. Evol. Computat.

Self Cite

105

View full text Add to dashboard Cite

Abstract-Knowledge transfer plays a vastly important role in solving multitask optimization problems (MTOPs). Many existing methods transfer task-specific knowledge such as the high-quality solution from one task to other tasks to enhance the optimization ability, which however may not work well or even have a negative effect if the tasks have very different task-specific knowledge. Hence, this paper proposes a meta-knowledge transfer-based differential evolution (MKTDE) algorithm by using a more general meta-knowledge transfer (MKT) method to solve MTOPs more efficiently. The meta-knowledge defined in this paper refers to the knowledge that can evolve task-specific knowledge during the evolutionary search. That is, the meta-knowledge is a kind of "knowledge of knowledge", which denotes the knowledge of "how to solve problem via evolution" and "the feature of evolving high-quality solution". The evolutionary search for solving different tasks can share common meta-knowledge even though these tasks involve heterogeneous data and have very different task-specific knowledge. Therefore, the MKT can associate the heterogeneous multi-source data of different tasks via transferring the meta-knowledge to help solve MTOPs more efficiently in a more general way. Moreover, to further enhance the MKTDE, two novel and efficient methods are proposed. One is a multiple populations for multiple tasks framework using a unified search space for making knowledge transfer flexibly. The other is an elite solution transfer method for achieving positive high-quality solution transfer. The superior performance of the proposed MKTDE is verified via extensive numerical experiments on both widely-used MTOP benchmark problems and real-world robot navigation problems, with comparisons with some state-of-the-art and the latest well-performing algorithms.

show abstract

Section: Discussionmentioning

confidence: 99%

A Meta-Knowledge Transfer-Based Differential Evolution for Multitask Optimization

Zhan

Tan

et al. 2022

IEEE Trans. Evol. Computat.

Self Cite

105

View full text Add to dashboard Cite

show abstract

“…Koziel [73] investigated a multi-fidelity optimization in which the computational model′s fidelity level can be adaptively modified. With simulation-based FE of different accuracy scales, Wu et al [74] developed a scale-adaptive FE (SAFE) approach for the crowdshipping scheduling application problem, which can strike a better balance between solution accuracy and computational cost. Li et al [75] proposed a surrogate-assisted multi-level evaluation method to reduce the expensive computational cost in optimizing the CNN hyperparameters.…”

Section: Multi-fidelity Substitutionmentioning

confidence: 99%

“…Multi-fidelity substitution SAFE [74] Expensive, single-objective A novel scale-adaptive fitness evaluation method…”

Section: Ieee Transactions On Evolutionary Computationmentioning

confidence: 99%

Evolutionary Computation for Expensive Optimization: A Survey

Zhan

Zhang

2022

Mach. Intell. Res.

Self Cite

View full text Add to dashboard Cite

Expensive optimization problem (EOP) widely exists in various significant real-world applications. However, EOP requires expensive or even unaffordable costs for evaluating candidate solutions, which is expensive for the algorithm to find a satisfactory solution. Moreover, due to the fast-growing application demands in the economy and society, such as the emergence of the smart cities, the internet of things, and the big data era, solving EOP more efficiently has become increasingly essential in various fields, which poses great challenges on the problem-solving ability of optimization approach for EOP. Among various optimization approaches, evolutionary computation (EC) is a promising global optimization tool widely used for solving EOP efficiently in the past decades. Given the fruitful advancements of EC for EOP, it is essential to review these advancements in order to synthesize and give previous research experiences and references to aid the development of relevant research fields and real-world applications. Motivated by this, this paper aims to provide a comprehensive survey to show why and how EC can solve EOP efficiently. For this aim, this paper firstly analyzes the total optimization cost of EC in solving EOP. Then, based on the analysis, three promising research directions are pointed out for solving EOP, which are problem approximation and substitution, algorithm design and enhancement, and parallel and distributed computation. Note that, to the best of our knowledge, this paper is the first that outlines the possible directions for efficiently solving EOP by analyzing the total expensive cost. Based on this, existing works are reviewed comprehensively via a taxonomy with four parts, including the above three research directions and the real-world application part. Moreover, some future research directions are also discussed in this paper. It is believed that such a survey can attract attention, encourage discussions, and stimulate new EC research ideas for solving EOP and related real-world applications more efficiently.

show abstract

“…When dealing with the MO-MTO problem (MO-MTOP), evolutionary computation (EC) methods are usually adopted, which leads to a promising research topic, i.e., evolutionary MO-MTO (denoted as EMO-MTO in the following contents) [13]. This is due to the fact that EC algorithms are powerful and efficient in tackling various complex optimization problems with different characteristics and difficulties [14][15][16][17]. The widely used EC algorithms include genetic algorithm (GA) [18][19][20], particle swarm optimization (PSO) [21][22][23], differential evolution (DE) [24][25][26][27], estimation of distribution algorithm (EDA) [28][29][30], ant colony optimization (ACO) [31][32][33], and evolutionary search (ES) [34,35].…”

Section: Introductionmentioning

confidence: 99%

Multi-objective multi-criteria evolutionary algorithm for multi-objective multi-task optimization

Wang

et al. 2022

Complex Intell. Syst.

Self Cite

View full text Add to dashboard Cite

Evolutionary multi-objective multi-task optimization is an emerging paradigm for solving multi-objective multi-task optimization problem (MO-MTOP) using evolutionary computation. However, most existing methods tend to directly treat the multiple multi-objective tasks as different problems and optimize them by different populations, which face the difficulty in designing good knowledge transferring strategy among the tasks/populations. Different from existing methods that suffer from the difficult knowledge transfer, this paper proposes to treat the MO-MTOP as a multi-objective multi-criteria optimization problem (MO-MCOP), so that the knowledge of all the tasks can be inherited in a same population to be fully utilized for solving the MO-MTOP more efficiently. To be specific, the fitness evaluation function of each task in the MO-MTOP is treated as an evaluation criterion in the corresponding MO-MCOP, and therefore, the MO-MCOP has multiple relevant evaluation criteria to help the individual selection and evolution in different evolutionary stages. Furthermore, a probability-based criterion selection strategy and an adaptive parameter learning method are also proposed to better select the fitness evaluation function as the criterion. By doing so, the algorithm can use suitable evaluation criteria from different tasks at different evolutionary stages to guide the individual selection and population evolution, so as to find out the Pareto optimal solutions of all tasks. By integrating the above, this paper develops a multi-objective multi-criteria evolutionary algorithm framework for solving MO-MTOP. To investigate the proposed algorithm, extensive experiments are conducted on widely used MO-MTOPs to compare with some state-of-the-art and well-performing algorithms, which have verified the great effectiveness and efficiency of the proposed algorithm. Therefore, treating MO-MTOP as MO-MCOP is a potential and promising direction for solving MO-MTOP.

show abstract

Surrogate-Assisted Hybrid-Model Estimation of Distribution Algorithm for Mixed-Variable Hyperparameters Optimization in Convolutional Neural Networks

Cited by 39 publications

References 62 publications

A Meta-Knowledge Transfer-Based Differential Evolution for Multitask Optimization

A Meta-Knowledge Transfer-Based Differential Evolution for Multitask Optimization

Evolutionary Computation for Expensive Optimization: A Survey

Multi-objective multi-criteria evolutionary algorithm for multi-objective multi-task optimization

Contact Info

Product

Resources

About