Deep Reinforcement Learning Based Active Queue Management for IoT Networks

Kim, Minsu; Jaseemuddin, Muhammad; Anpalagan, Alagan

doi:10.1007/s10922-021-09603-x

Cited by 29 publications

(10 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The problem of modeling, managing and optimizing resources in a heterogeneous communication network is a very challenging engineering problem because of its inherent complexity ( [1], [2], [31]- [33]). Indeed, one of the most difficult challenges to be addressed to apply optimization techniques to the management of such complex networks is to derive predictive models of the queues of the switch behaviour ( [34]- [39]). Numerous studies have also been conducted to maximize the performance of the controller and OpenFlow switch of SDNs.…”

Section: Related Workmentioning

confidence: 99%

Modeling and Control of Priority Queueing in Software Defined Networks via Machine Learning

et al. 2022

View full text Add to dashboard Cite

Software Defined Networking (SDN) is a new architectural paradigm that enables programmable control of a network to make it more flexible and easier to manage. SDN architectures decouple control and forwarding functionalities, and enable switches and routers to be remotely configurable/programmable in run-time by a controller. Modeling and optimization of such modern heterogeneous network infrastructures are key factors to achieve better performance, e.g. in terms of traffic flow improvement while reducing bandwidth allocation. Identifying an accurate model of a network device in SDNs (e.g., a switch or a router) is crucial in order to apply advanced techniques such as Model Predictive Control (MPC). However, such a problem is very challenging due to non-linearities and unavailability of internal variables measurements in real devices. To this aim, a promising direction is given by an appropriate integration of System Identification and Machine Learning techniques to obtain predictive models using historical data collected from the network thanks to the SDN paradigm. In this paper we apply a novel data-driven methodology to learn accurate models of the dynamical input-output behavior of a network's switch device by appropriately combining AutoRegressive eXogenous (ARX) model identification with Regression Trees (RTs) and Random Forests (RFs). The advantage of such model is that it can be directly used to apply MPC (which just requires Quadratic Programming to be solved) to optimally control the queues' bandwidth of the switch ports within the SDN paradigm. We validate our approach on an experimental emulation setup using the Mininet network emulator environment and a real dataset obtained from measurements of an Italian Internet Service Provider (Sonicatel S.r.l.). To this aim, we first develop a model of a real network switch, then implement MPC using the RYU controller, and finally demonstrate the benefits of the proposed dynamic queueing control methodology in terms of packet losses reduction and bandwidth saving, i.e. in terms of improvement of the Quality of Service.

show abstract

Section: Related Workmentioning

confidence: 99%

Modeling and Control of Priority Queueing in Software Defined Networks via Machine Learning

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Based on the idea of the ARED algorithm, the target queue length, target, is set with a value range of [min th +0.4 (max th − min th ), min th +0.6 (max th − min th )], and the relationship between target and Qavg is used to adaptively adjust maxp [21]. If Qavg is in the vicinity of min th and Qavg < target, the congestion adjustment is too active, and hence, the value of maxp must be decreased; if Qavg is in the vicinity of max th and Qavg > target, the congestion adjustment is too conservative, and the value of maxp must be increased [22]. Let maxp + and maxp − represent the maximum drop probabilities obtained after a radical increase and a conservative decrease, respectively, which are expressed as follows:…”

Section: ) Packet Drop Probability Based On the Cubic Functionmentioning

confidence: 99%

Queue Management Algorithm for Satellite Networks Based on Traffic Prediction

2022

View full text Add to dashboard Cite

In connexion with the effect of the self-similar characteristic of satellite network service traffic on queueing performance, a prediction model with optimised triple exponential smoothing is first established in this paper. This model performs network traffic prediction based on the dynamic triple exponential smoothing model and optimises the smoothing coefficient of the model through the differential evolution algorithm; a cubic function based on traffic prediction is further proposed to improve the adaptive random early detection (ARED) queue management algorithm. Based on the network traffic prediction results and the ARED, this algorithm uses the cubic function to perform nonlinear processing on the packet drop probability function. The simulation results show that the prediction model with optimised triple exponential smoothing has a high prediction accuracy, and the improved ARED algorithm based on the cubic function of traffic prediction in the presence of data bursts in self-similar traffic. It can effectively reduce the packet loss rate and improve the throughput, so as to better control the network congestion caused by self-similar traffic in satellite network.INDEX TERMS Dynamic triple exponential smoothing model, differential evolution algorithm, cubic function, adaptive random early detection.

show abstract

“…The goal of MPC is to minimize the cost function (23). In order to find the optimal control input, equation ( 23) can be written as:…”

Section: B Controller Designmentioning

confidence: 99%

“…[22] presented a solution that tackles the challenges of tuning the active queue management parameters for inter-domain congestion control scenarios by using an artificial neural network. [23] adopted deep reinforcement learning technique to achieve the trade-off between queuing delay and throughput. However, most of the methods above rely on the network system model, and require a large amount of computation and storage space.…”

mentioning

confidence: 99%

Data-Driven Congestion Control of Micro Smart Sensor Networks for Transparent Substations

Wang¹,

Qiu²,

Li³

2021

IEEE Access

View full text Add to dashboard Cite

Micro smart sensors and sensor networks are the key bases for building a fully visible, perceptible and controllable transparent substation in power grid. However, the massive deployment of micro smart sensors often leads to network congestion and directly affects the quality of information transmission. In practice, control design for micro sensor networks is often disturbed by factors such as nonlinearity, time delay and time-varying parameters, and the mathematical model is inaccurate. To solve these problems, this paper proposes a Koopman operator based data-driven congestion control scheme without using any system model information, based on model predictive control (MPC) and extended state observer (ESO). Firstly, according to the Koopman operator theory, a data-driven linear Koopman model for micro smart sensor networks is derived, only using the input/output data. Then an ESO based MPC scheme is designed to obtain the optimal packet dropping probability in the micro smart sensor node buffer area. Specifically, ESO performs real-time online estimation of the modeling errors, such as time-varying parameters, time delay and the external disturbances. Then the estimated modeling errors are viewed as total disturbances and are compensated in MPC, so that the queue length in the node buffer can quickly stabilize at the set value. Extensive simulations are conducted to verify the accuracy of Koopman model, and the effectiveness of the proposed control system design.INDEX TERMS Koopman operator, network congestion, micro smart sensor network, extended state observer I. INTRODUCTION

show abstract

Deep Reinforcement Learning Based Active Queue Management for IoT Networks

Cited by 29 publications

References 31 publications

Modeling and Control of Priority Queueing in Software Defined Networks via Machine Learning

Modeling and Control of Priority Queueing in Software Defined Networks via Machine Learning

Queue Management Algorithm for Satellite Networks Based on Traffic Prediction

Data-Driven Congestion Control of Micro Smart Sensor Networks for Transparent Substations

Contact Info

Product

Resources

About