Applying reinforcement learning towards automating resource allocation and application scalability in the cloud

Abstract

SUMMARYPublic Infrastructure as a Service (IaaS) clouds such as Amazon, GoGrid and Rackspace deliver computational resources by means of virtualisation technologies. These technologies allow multiple independent virtual machines to reside in apparent isolation on the same physical host. Dynamically scaling applications running on IaaS clouds can lead to varied and unpredictable results due to the performance interference effects associated with co-located virtual machines. Determining appropriate scaling policies in a dynamic non-stationary environment is non-trivial. One principle advantage exhibited by IaaS clouds over their traditional hosting counterparts is the ability to scale resources on-demand. However a problem arises concerning resource allocation as to which resources should be added and removed when the underlying performance of the resource is in a constant state of flux. Decision theoretic frameworks such as Markov Decision Processes are particularly suited to decision making under uncertainty. By applying a temporal difference reinforcement learning algorithm known as Q-learning, optimal scaling policies can be determined. Additionally reinforcement learning techniques typically suffer from curse of dimensionality problems, where the state space grows exponentially with each additional state variable. To address this challenge we also present a novel parallel Q-learning approach aimed at reducing the time taken to determine optimal policies whilst learning online.

Fuzzy Self-Learning Controllers for Elasticity Management in Dynamic Cloud Architectures

“…ii A cloud controller reasoning engine (RobusT2Scale [18]) implemented in Matlab. 2 iii A cloud-based application framework (ElasticBench) implemented with Microsoft .NET technologies (.NET framework 4 and Azure SDK 2.5). 3 iv The integration between these three components by software connectors (cf.…”

Autonomic resource provisioning for cloud-based software

“…However, the prediction accuracy highly depends on the number observations and the interval [18]. Reinforcement learning (e.g., [42]) enable learning elasticity policies from observations. However, it requires long learning, which is only applicable for stable workloads.…”

Using Machine Learning in Trace-driven Energy-Aware Simulations of High-Throughput Computing Systems

“…Barrett et al [6] consider a Q-learning approach to the auto-scaling of cloud applications deployment. Seeking to address the dimensionality issues associated with Reinforcement Learning approaches by adopting a hybrid approach.…”