Modeling on virtual network embedding using reinforcement learning

Abstract

It is well known that virtual network (VN) embedding (VNE) aims to solve how to efficiently allocate physical resources to a VN. However, this issue has been proved to be an NP-hard problem. Besides, as most of the existing approaches are based on heuristic algorithms, which is easy to fall into local optimal. To address the challenge, we formalize the problem as a mixed integer programming problem and propose a novel VNE method based on reinforcement learning in this article. And to solve the problem, we introduce a pointer network to generate virtual node mapping strategies through an attention mechanism, and design a reward function related to link resource consumption to build the connection between node mapping and link mapping stages of VNE. In addition, we present a policy gradient optimization mechanism to leverage the reward information obtained from the sampled solutions, and design an active search based process to automatically update the parameters of the neural network and to obtain near-optimal embedding solution. The experimental results show that the proposed method can improve the performance in average physical node utilization and long-term revenue to cost ratio comparing than that of the existing models. K E Y W O R D S attention, pointer network, policy gradient, reinforcement learning, virtual network embedding 1 INTRODUCTION Network virtualization technology can allow multiple heterogeneous network architectures to coexist on a shared physical substrate network (SN). 1 In network virtualization, the role of traditional Internet service providers (ISPs) is separated into two independent parts: infrastructure providers (InPs), who manage the physical infrastructure, and service providers (SPs), who provide services by renting virtual network (VN) using resources from InPs. 2 In such a way, enterprises are increasingly deploying services 3 to the Cloud with the development of network function virtualization (NFV) 4 and software defined networking (SDN). 5 The major challenges of network virtualization are how to effectively allocate SN resources to VNs as well as increase the utilization and revenue of InPs, which is known as the virtual network embedding (VNE) problem. 6 The VNE problem mainly includes two parts: node mapping stage and link mapping stage. It has been proven to be NP-hard. 7 Many exact and heuristic algorithms 8-13 have been proposed to solve the VNE problem in recent years. Although exact solutions can achieve high resource utilization rate of the SN, they require exponential time-consuming and therefore are not practical. In addition, most of the heuristic algorithms for VNE consider link mapping and node mapping separately. They did not consider the inner practical connection between the two stages and may result in suboptimal solutions. Furthermore, they are unable to leverage historical experience to optimize themselves automatically.

Virtual Network Embedding Based on Hierarchical Cooperative Multiagent Reinforcement Learning

“…In [29], the authors propose an RL-based VNE algorithm using a pointer network model [43]. The algorithm employs the attention mechanism to focus on a specific substrate node, and the pointer network model comprises an encoderdecoder that takes the substrate node's features as input.…”

Reinforcement Learning-Based Slice Isolation Against DDoS Attacks in Beyond 5G Networks

“…The features of the SNs and VNs construct the processes image. Wang et al [26] proposed a RL model, called PNVNE (Pointer Network VNE), for solving VNE problems, which utilizes an attention mechanism that selects the most suitable substrate nodes, hence, the model can focus on them. Troia et al [27] proposed an Advantage Actor-Critic RL (A2CRL) model for mapping slice requests in a 5G network on the SNs, as a special form of VNE problems.…”

“…Then, we have compared the performance of SIRL against nine of the existing RL models, which are CDRL [17], RDAM [18], VNEQS [20], MLRL [21], DRLVNE [22], GCNNRL [23], A3CGCN [24], DeepViNE [25], PNVNE [26], and A2CRL [27]. Since our focus is on the features and their ability to model the environment, we have considered the same reward function for all the simulated models to remove the impact of the rewards on the model's performance.…”

Network Slicing Resource Mapping Algorithm Oriented to Delay and Reliability Awareness

“…The network performance can be recovered faster by means of backup, but the backup will occupy more resources and increase the mapping cost. In [18,19], the author quantifies the failure of the underlying equipment as a reliability evaluation index, and proposes a slice mapping algorithm for different business types, which not only ensures the reliability of slices, but also meets the diversified needs of the business. In [20], the author comprehensively considers the underlying topology, reliability evaluation of a single node and the reliability evaluation of the adjacent environment of the node as the reliability indicators of the node.…”