Optimizing neural network hyperparameters with Gaussian processes for dialog act classification

Abstract

Systems based on artificial neural networks (ANNs) have achieved state-of-the-art results in many natural language processing tasks. Although ANNs do not require manually engineered features, ANNs have many hyperparameters to be optimized. The choice of hyperparameters significantly impacts models' performances. However, the ANN hyperparameters are typically chosen by manual, grid, or random search, which either requires expert experiences or is computationally expensive. Recent approaches based on Bayesian optimization using Gaussian processes (GPs) is a more systematic way to automatically pinpoint optimal or near-optimal machine learning hyperparameters. Using a previously published ANN model yielding state-of-the-art results for dialog act classification, we demonstrate that optimizing hyperparameters using GP further improves the results, and reduces the computational time by a factor of 4 compared to a random search. Therefore it is a useful technique for tuning ANN models to yield the best performances for natural language processing tasks.

Machine learning in airline crew pairing to construct initial clusters for dynamic constraint aggregation

“…More specifically, we optimize the hyperparameters listed in Table 1 [37] with an implementation of Gaussian process-based Bayesian optimization provided by the GPyOpt Python library version 1.2.1 [38]. Bayesian optimization constructs a probabilistic model of the function mapping from hyperparameter settings to the model performance and provides a systematic way to explore the space more efficiently [39].…”

Hyper-parameter Optimisation by Restrained Stochastic Hill Climbing

“…HPO is the process of optimising a loss function over a graph-structured configuration space [4]; the aim is to maximise or minimise a given function [39]. Determining appropriate values for the hyperparameters is fundamental in finding an optimal solution, however, it is a frustratingly difficult task [18,10,8,9] and the performance of NNs crucially depend on the hyper-parameters used [9,6]. For example, the internal structure is a key factor in determining the efficiency of the NN [27].…”

Fees and governance: Towards sustainability in water resources management at schools in post-apartheid South Africa

“…There has been a substantial amount of research dedicated to urban water demand management, which is particularly essential in developing countries as they often suffer from high rates of urbanisation. The majority of existing research on urban water demand management have focused on the residential sector, for example, demand forecasting (Adamowski et al, 2012;Bougadis et al, 2005;Donkor et al, 2012;Ghiassi et al, 2017;Ren and Li, 2016), demand modelling (Gurung et al, 2014;Jacobs and Haarhoff, 2004), general demand management (Kenney et al, 2008), and water usage management interventions (Datta et al, 2015;Dernoncourt and Lee, 2016;Fielding et al, 2012). There is limited research on the water demand in the non-residential or educational sectors despite the fact that these sectors can be high water consumers (Sánchez-Torija et al, 2017).…”