Lamarckian Evolution of Convolutional Neural Networks

Prellberg, Jonas; Krämer, Oliver

doi:10.1007/978-3-319-99259-4_34

Cited by 14 publications

(6 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [5], authors propose a population-based method that renders, at each generation, a set of architectures generated using local transformation. They use Lamarckian inheritance [15] to permit faster architecture evaluation. Their work, however, uses complex operators which makes it difficult to adapt to other tasks.…”

Section: B Multi-objective Neural Architecture Searchmentioning

confidence: 99%

An Alternative Pareto-Based Approach to Multi-Objective Neural Architecture Search

Zouambi,

Dhaenens,

Jacques

2023

2023 IEEE Congress on Evolutionary Computation (CEC)

View full text Add to dashboard Cite

Neural architecture search (NAS) is a field that automates the architecture design of neural networks. NAS can be modeled as an optimization problem. It describes a space of possible architectures and looks for the most performing one. NAS, however, is not limited to finding the most taskaccurate neural network. It can consider other objectives during the search to meet different demands and requirements (model size, latency, energy consumption, etc.). Several methods were proposed for multi-objective NAS. Most of them are either based on the scalarization of objectives, or use a Pareto-based approach. Methods based on scalarization require preference weighting between objectives and can suffer from suboptimality, while the Pareto-based methods found in NAS usually require complex operators and many parameters to tune. The goal of our work is to offer an alternative Pareto-based method that solves the above issues. In this paper, we first present a formulation of the NAS problem as a multi-objective optimization (MO) problem. We then design a dominance-based multi-objective local search (DMLS) to solve it. Unlike other NAS methods, our work uses a simple encoding, few parameters, and does not require preference weighting of objectives. To assess its performance, we evaluate this algorithm on a specialized multi-objective NAS benchmark to optimize both accuracy and network complexity. We compare it to state-of-the-art MO methods of this benchmark. Results show that our method finds significantly more Pareto optimal solutions than NGSA-II and overpasses single-objective local search for the same evaluation budget. We conclude that DMLS provides a more practical MO approach for NAS while providing superior performances.

show abstract

Section: B Multi-objective Neural Architecture Searchmentioning

confidence: 99%

An Alternative Pareto-Based Approach to Multi-Objective Neural Architecture Search

Zouambi,

Dhaenens,

Jacques

2023

2023 IEEE Congress on Evolutionary Computation (CEC)

View full text Add to dashboard Cite

show abstract

“…Furthermore, Johner et al [57] use a ranking function to choose individuals by rank. A selection trick termed as niching is used in [67], [126] to avoid stacking into local optima. This trick allows offspring worse than parent for several generations until evolving to a better one.…”

Section: B Selection Strategymentioning

confidence: 99%

“…Another way is reducing the population dynamically. For instance, Fan et al [122] use the (µ + λ) evolution Weight inheritance [19], [36], [38], [40], [55], [56], [62], [67], [77], [78], [84], [86], [88], [105], [109], [111], [120], [121], [125], [132] Early stopping policy [21], [32], [36], [42], [43], [62], [65], [71], [79], [80], [83], [86], [86], [95], [98], [103], [108], [112], [114], [125], [145] Reduced training set [46], [84], [124], [129], [188] Reduced population [98],…”

Section: Shorten the Evaluation Timementioning

confidence: 99%

“…The column "CIFAR-10" and "CIFAR-100" denote the error rate of each method on the corresponding datasets respectively. Furthermore, the "GPU Days" denotes the total search time of each method, it can be calculated by the Equation 9 Image classification [9], [19]- [21], [26], [29]- [32], [36], [38]- [40], [42]- [44], [49], [50], [56]- [58], [61], [62], [67], [68], [70], [71], [71], [74], [78], [80], [83], [84], [88]- [90], [92], [95], [96], [98], [99], [101], [102], [104], [105],…”

Section: B Comparison On Cifar-10 and Cifar-100mentioning

confidence: 99%

See 1 more Smart Citation

A Survey on Evolutionary Neural Architecture Search

Liu,

Sun,

Xue

et al. 2020

Preprint

View full text Add to dashboard Cite

Deep Neural Networks (DNNs) have achieved great success in many applications, such as image classification, natural language processing and speech recognition. The architectures of DNNs have been proved to play a crucial role in its performance. However, designing architectures for different tasks is a difficult and time-consuming process of trial and error. Neural Architecture Search (NAS), which received great attention in recent years, can design the architecture automatically. Among different kinds of NAS methods, Evolutionary Computation (EC) based NAS methods have recently gained much attention and success. Unfortunately, there is not a comprehensive summary of the EC-based methods. This paper reviews 100+ papers of EC-based NAS methods in light of the common process. Four steps of the process have been covered in this paper including population initialization, population operators, evaluation and selection. Furthermore, current challenges and issues are also discussed to identify future research in this field.

show abstract

“…Liu et al [29] suggested using a genetic algorithm for evolving individuals using mutation by adding, removing, or editing edges in a computation graph which can be translated into a convolutional neural network. Finally, first Kramer [30] and later Prellberg and Kramer [31] have presented an approach based on an evolutionary algorithm that relies only on the mutation operator and have introduced a mechanism to support parameters inheritance, so that descendants during the evolutionary process do not need to learn weights from scratch. Assuno et al [32] have presented DENSER, a work where a multi-level encoding of candidate solutions allow for the optimization of the topology of the network and the activation functions, with authors claiming that it can be used also to evolve the hyperparameters of the learning process as well as of the data augmentation stage.…”

Section: Complexitymentioning

confidence: 99%

Hybridizing Evolutionary Computation and Deep Neural Networks: An Approach to Handwriting Recognition Using Committees and Transfer Learning

2019

View full text Add to dashboard Cite

Neuroevolution is the field of study that uses evolutionary computation in order to optimize certain aspect of the design of neural networks, most often its topology and hyperparameters. The field was introduced in the late-1980s, but only in the latest years the field has become mature enough to enable the optimization of deep learning models, such as convolutional neural networks. In this paper, we rely on previous work to apply neuroevolution in order to optimize the topology of deep neural networks that can be used to solve the problem of handwritten character recognition. Moreover, we take advantage of the fact that evolutionary algorithms optimize a population of candidate solutions, by combining a set of the best evolved models resulting in a committee of convolutional neural networks. This process is enhanced by using specific mechanisms to preserve the diversity of the population. Additionally, in this paper, we address one of the disadvantages of neuroevolution: the process is very expensive in terms of computational time. To lessen this issue, we explore the performance of topology transfer learning: whether the best topology obtained using neuroevolution for a certain domain can be successfully applied to a different domain. By doing so, the expensive process of neuroevolution can be reused to tackle different problems, turning it into a more appealing approach for optimizing the design of neural networks topologies. After evaluating our proposal, results show that both the use of neuroevolved committees and the application of topology transfer learning are successful: committees of convolutional neural networks are able to improve classification results when compared to single models, and topologies learned for one problem can be reused for a different problem and data with a good performance. Additionally, both approaches can be combined by building committees of transferred topologies, and this combination attains results that combine the best of both approaches.

show abstract

Lamarckian Evolution of Convolutional Neural Networks

Cited by 14 publications

References 13 publications

An Alternative Pareto-Based Approach to Multi-Objective Neural Architecture Search

An Alternative Pareto-Based Approach to Multi-Objective Neural Architecture Search

A Survey on Evolutionary Neural Architecture Search

Hybridizing Evolutionary Computation and Deep Neural Networks: An Approach to Handwriting Recognition Using Committees and Transfer Learning

Contact Info

Product

Resources

About