Online-Autotuning in the Presence of Algorithmic Choice

Pfaffe, Philip; Tillmann, Martin; Walter, Sigmar; Tichy, Walter F.

doi:10.1109/ipdpsw.2017.28

Cited by 11 publications

(9 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are several works that rely on the Nelder-Mead algorithm for optimisation [19][20][21][22]. Koenigstein et al [19] adopt the Nelder-Mead direct search to optimise more than twenty hyperparameters.…”

Section: Related Workmentioning

confidence: 99%

“…The authors adopt Nelder-Mead to identify the optimal hyperparameter initialisation. Pfaffe et al [22] present an online auto tuning algorithm for string matching algorithms. It uses the e-greedy policy, a well-known reinforcement learning technique, to select the algorithm to be used in each iteration and adopts Nelder-Mead to tune the parameters until a pre-defined user criterion is met, e.g., number of tuning iterations.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Hyperparameter self-tuning for data streams

et al. 2021

View full text Add to dashboard Cite

The number of Internet of Things devices generating data streams is expected to grow exponentially with the support of emergent technologies such as 5G networks. The online processing of these data streams therefore requires the design and development of suitable machine learning algorithms, able to learn online, as data is generated. Like their batch-learning counterparts, streambased learning algorithms require careful hyperparameter settings. However, this problem is exacerbated in online learning settings, especially with the occurrence of concept drifts, that frequently require the reconfiguration of hyperparameters. In this article, we present SSPT, an extension of the Self Parameter Tuning (SPT) optimisation algorithm for data streams. We apply the Nelder-Mead algorithm to dynamically-sized samples, converging to optimal settings in a single-pass over data, while using a relatively small number of hyperparameter configurations. In addition, our proposal automatically readjusts hyperparameters when concept drift occurs. To assess the effectiveness of SSPT, the algorithm is evaluated with three different machine learning problems: recommendation, regression, and classification. Experiments with well-known data sets show that the proposed algorithm can outperform previous hyperparameter tuning efforts by human experts. Results show that SSPT converges significantly faster and

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Hyperparameter self-tuning for data streams

et al. 2021

View full text Add to dashboard Cite

show abstract

“…In machine learning, the ability to select appropriate features, work flows, machine learning paradigms, algorithms, and their hyper-parameters requires expert knowledge [13]. The few contributions found in the literature addressing this progressive automation of machine learning or auto-ML include tools [1,27,9], model selection algorithms [7,6], hyper-parameter optimisation algorithms [18,10,24] and Nelder-Mead optimisation solutions [16,8,25].…”

Section: Related Workmentioning

confidence: 99%

“…In terms of automatic hyperparameter selection, there are several different techniques: (i ) particle swarm optimisation [7], which is flexible and can be applied to ensemble models [6]; (ii ) grid search [18], which minimises the estimated error until converges on a local minima; (iii) gradient-based search, e.g., Stochastic Gradient Descent (SGD), which converges to an optimal solution [24]; and (iv ) Nelder-Mead direct search, which relies on heuristics to optimise model parameters [16] or tensor based models [8]. The Nelder-Mead algorithm has been used together with exponentially decay centrifugal forces to improve the results at the cost of the number of iterations needed to converge [15] as well as with reinforcement techniques (e-greedy) to select the best model of each iteration [25].…”

Section: Related Workmentioning

confidence: 99%

Self Hyper-parameter Tuning for Stream Recommendation Algorithms

Veloso

Gama

Malheiro

et al. 2019

Communications in Computer and Information Science

View full text Add to dashboard Cite

E-commerce platforms explore the interaction between users and digital content-user generated streams of events-to build and maintain dynamic user preference models which are used to make meaningful recommendations. However, the accuracy of these incremental models is critically affected by the choice of hyper-parameters. So far, the incremental recommendation algorithms used to process data streams rely on human expertise for hyper-parameter tuning. In this paper we apply our Self Hyper-Parameter Tuning (SPT) algorithm to incremental recommendation algorithms. SPT uses the Nelder & Mead optimisation algorithm to perform hyper-parameter tuning. It creates three models with different hyper-parameters, assesses them at dynamic size intervals and applies the Nelder & Mead operators to update their hyperparameters until they converge. The main contribution of this work is the adaptation of the SPT method to incremental matrix factorisation recommendation algorithms. The proposed method was evaluated with well-known recommendation data sets. The results show that SPT systematically improves data stream recommendations.

show abstract

“…Auto-tuning and input aware techniques [13,16] are recently used to address the problem of performance portability on different data-driven applications [11,23,32]. An interesting approach extends such techniques in the presence of multiple algorithmic choice [42]. However, their on-line solution is suitable when a specific routine is called multiple times.…”

Section: Related Workmentioning

confidence: 99%

A model-driven approach for a new generation of adaptive libraries

Cianfriglia,

Vella,

Nugteren

et al. 2018

Preprint

View full text Add to dashboard Cite

Efficient high-performance libraries often expose multiple tunable parameters to provide highly optimized routines. These can range from simple loop unroll factors or vector sizes all the way to algorithmic changes, given that some implementations can be more suitable for certain devices by exploiting hardware characteristics such as local memories and vector units. Traditionally, such parameters and algorithmic choices are tuned and then hard-coded for a specific architecture and for certain characteristics of the inputs. However, emerging applications are often data-driven, thus traditional approaches are not effective across the wide range of inputs and architectures used in practice. In this paper we present a new adaptive framework for data-driven applications which uses a predictive model to select the optimal algorithmic parameters by training with synthetic and real datasets. We demonstrate the effectiveness on a BLAS library and specifically on its matrix multiplication routine. We present experimental results for two GPU architectures, and show significant performance gains of up to 3x (on a high-end NVIDIA Pascal GPU) and 2.5x (on an embedded ARM Mali GPU) when compared to a traditionally optimized library.

show abstract

Online-Autotuning in the Presence of Algorithmic Choice

Cited by 11 publications

References 16 publications

Hyperparameter self-tuning for data streams

Hyperparameter self-tuning for data streams

Self Hyper-parameter Tuning for Stream Recommendation Algorithms

A model-driven approach for a new generation of adaptive libraries

Contact Info

Product

Resources

About