HyperNOMAD

Abstract

The performance of deep neural networks is highly sensitive to the choice of the hyperparameters that define the structure of the network and the learning process. When facing a new application, tuning a deep neural network is a tedious and time-consuming process that is often described as a “dark art.” This explains the necessity of automating the calibration of these hyperparameters. Derivative-free optimization is a field that develops methods designed to optimize time-consuming functions without relying on derivatives. This work introduces the HyperNOMAD package, an extension of the NOMAD software that applies the MADS algorithm [7] to simultaneously tune the hyperparameters responsible for both the architecture and the learning process of a deep neural network (DNN). This generic approach allows for an important flexibility in the exploration of the search space by taking advantage of categorical variables. HyperNOMAD is tested on the MNIST, Fashion-MNIST, and CIFAR-10 datasets and achieves results comparable to the current state of the art.

Tuning a variational autoencoder for data accountability problem in the Mars Science Laboratory ground data system

“…This section describes ∆-MADS, a hybrid algorithm that mixes the local search of HyperNOMAD [24,25] with the global exploration scheme of ∆-DOGS [15]. ∆-MADS is designed to solve derivativefree optimization problems formulated as follows:…”

“…Bayesian methods offer a more sophisticated alternative by either constructing a model over the objective function f in the case of a Gaussian processes [30] or random forests [16], or over the distribution of the good and bad configurations in the case of tree parzen estimators [13], by using the previously evaluated points. Other approaches were tested such as reinforcement learning [10,32] which is successfully used to find the appropriate architecture of convolutional neural networks, and more recently the HyperNOMAD [24,25] software, based on the mesh adaptive direct search (MADS) algorithm [8], was able to yield good results when optimizing both the architecture and the training hyperparameters simultaneously. The main drawback of this software is its lack of global exploration strategy.…”

Use of static surrogates in hyperparameter optimization

“…The open-source library HyperNOMAD [18,19] is designed as an adaptation of the NOMAD software [20] to optimize the hyperparameters of deep neural networks as formulated in (1). This package allows searching for both the archi-tecture and the convolutional network's training regime for a specific dataset.…”

“…The properties of DFO methods explain their popularity in the context of HPO of deep neural networks where they are often included in specialized libraries such as Hyperopt [6] or Oríon [7]. Similarly, the HyperNOMAD toolbox [18,19] is developed as an adaptation of MADS to simultaneously optimize the architecture and the training phase of a CNN for a given dataset as expressed in (1). This open source library was shown to have a competitive performance against other popular approaches such as Bayesian optimization [4] or a random search [5] on the MNIST [21], Fashion-MNIST [29] and datasets.…”

“…Figure 4: Comparison between the original HyperNOMAD[18,19] and four variants that implement a strategy to sort the new candidates in order to evaluate the most promising first. Strategies R 1 and R 2 train on a small epoch budget while strategies R 3 and R 4 train on a subset of the data.…”