A deep recurrent Q network towards self‐adapting distributed microservice architecture

Abstract

One desired aspect of microservices architecture is the ability to self-adapt its own architecture and behaviour in response to changes in the operational environment. To achieve the desired high levels of self-adaptability, this research implements the distributed microservices architectures model, as informed by the MAPE-K model. The proposed architecture employs a multi adaptation agents supported by a centralised controller, that can observe the environment and execute a suitable adaptation action. The adaptation planning is managed by a deep recurrent Q-network (DRQN). It is argued that such integration between DRQN and MDP agents in a MAPE-K model offers distributed microservice architecture with self-adaptability and high levels of availability and scalability. Integrating DRQN into the adaptation process improves the effectiveness of the adaptation and reduces any adaptation risks, including resources over-provisioning and thrashing. The performance of DRQN is evaluated against deep Q-learning and policy gradient algorithms including: i) deep q-network (DQN), ii) dulling deep Q-network (DDQN), iii) a policy gradient neural network (PGNN), and iv) deep deterministic policy gradient (DDPG). The DRQN implementation in this paper manages to outperform the above mentioned algorithms in terms of total reward, less adaptation time, lower error rates, plus faster convergence and training times. We strongly believe that DRQN is more suitable for driving the adaptation in distributed services-oriented architecture and offers better performance than other dynamic decision-making algorithms. Index TermsService oriented architecture, self-adaptive architectures, reinforcement learning, Q-learning algorithms, deep Q-Learning networks, recurrent Q-learning networks, policy approximation, multi agents environment.

Microservice‐driven privacy‐aware cross‐platform social relationship prediction based on sequential information

“…According to the literature, 53 it can be known that over time, users are able to notice event or people that has a far‐reaching influence on them, so as to form their unique preferences. To extract more comprehensive users' preference information, we increase the vertical depth in the r‐GRU network 54 by exploiting its scalability. In the middle part of Figure 4, $$$ H $$$ and $$$ T $$$ represent the vertical depth in the r‐GRU network and the number of time slices 55 (i.e., the horizontal depth in the r‐GRU network), respectively.…”

A Geometric Approach to Real-time Quality of Experience Prediction in Volatile Edge Networks

“…Magableh et al [6] illustrate the use of a Deep Recurrent Q-Network (DRQN) for an effective self-adaptive service architecture, although the algorithm is not designed for decentralization and can not be used in real-time, unlike the solution presented in this paper. Lu et al [7] use a solution based on double dueling Deep Q-networks (DQN) to determine optimal offloading policies in the edge.…”

Scalable Microservice Forensics and Stability Assessment Using Variational Autoencoders

“…Meng et al propose detecting variance in application call trees as a strategy to detect microservice performance anomalies [12]. Magableh uses deep Q-learning networks (DQNs) to build self-adaptive agents for performance optimization [11].…”