On the use of hybrid reinforcement learning for autonomic resource allocation

Abstract

Reinforcement Learning (RL) provides a promising new approach to systems performance management that differs radically from standard queuing-theoretic approaches making use of explicit system performance models. In principle, RL can automatically learn high-quality management policies without an explicit performance model or traffic model, and with little or no built-in system specific knowledge. In our original work (Das, R.) we showed the feasibility of using online RL to learn resource valuation estimates (in lookup table form) which can be used to make high-quality server allocation decisions in a multi-application prototype Data Center scenario. The present work shows how to combine the strengths of both RL and queuing models in a hybrid approach, in which RL trains offline on data collected while a queuing model policy controls the system. By training offline we avoid suffering potentially poor performance G. Tesauro ( ) · R. Das in live online training. We also now use RL to train nonlinear function approximators (e.g. multi-layer perceptrons) instead of lookup tables; this enables scaling to substantially larger state spaces. Our results now show that, in both openloop and closed-loop traffic, hybrid RL training can achieve significant performance improvements over a variety of initial model-based policies. We also find that, as expected, RL can deal effectively with both transients and switching delays, which lie outside the scope of traditional steady-state queuing theory.

Self-Tuning Transactional Data Grids: The Cloud-TM Approach

“…In [29], a model based on queuing theory is corrected at runtime by exploiting online reinforcement learning to determine the batching level delivering lowest latencies for a Total-Order based broadcast primitive. A similar approach is undertaken in [30], where the target is optimizing resource provisioning in a distributed application and the online learner is based on Q-learning. In [31], [32] analytical models are complemented at runtime by decision tree regressors, in the former case with the purpose of optimizing the global multiprogramming level for distributed transactional applications, in the latter to allow a continuous validation and correction of difference performance predictors in a data center.…”

A Learning Architecture for Scheduling Workflow Applications in the Cloud

“…While resource allocation is more concerned with low level scheduling of tasks at the virtual machine level, the parallels between them still merit their inclusion. Tesauro investigated the use of a hybrid reinforcement learning technique for autonomic resource allocation [15]. He applied this research to optimizing server allocation in data centers.…”

Synthesis of application-level utility functions for autonomic self-assessment

“…To efficiently calculate those trajectories, we could again leverage machine learning techniques, in particular Reinforcement Learning (RL). RL has been successfully used for Autonomic Computing in the recent past to synthesize policies for decision-making, aiming once again at self-optimization [27] [28]. A key concept in RL is "reward", a scalar that valuates the observed consequence of a decision, and which the learning agent making decisions aims to maximize.…”