Digital Twin Assisted Risk-Aware Sleep Mode Management Using Deep Q-Networks

Abstract

Base stations (BSs) are the most energy-consuming segment of mobile networks. To reduce BS energy consumption, different components of BSs can sleep when BS is not active. According to the activation/deactivation time of the BS components, multiple sleep modes (SMs) are defined in the literature. In this study, we model the problem of BS energy saving utilizing multiple sleep modes as a sequential Markov decision process (MDP) and propose an online traffic-aware deep reinforcement learning approach to maximize the long-term energy saving. However, there is a risk that BS is not sleeping at the right time and incurs large delays to the users. To tackle this issue, we propose to use a digital twin model to encapsulate the dynamics underlying the investigated system and estimate the risk of decision-making (RDM) in advance. We define a novel metric to quantify RDM and predict the performance degradation. The RDM calculated by DT is compared with a tolerable threshold set by the mobile operator. Based on this comparison, BS can decide to deactivate the SMs, re-train when needed to avoid taking high risks, or activate the SMs to benefit from energy savings. For deep reinforcement learning, we use long-short term memory (LSTM), to take into account the long and short-term dependencies in input traffic, and approximate the Q-function. We train the LSTM network using the experience replay method over a real traffic data set obtained from an operator's BS in Stockholm. The data set contains data rate information with very coarse-grained time granularity. Thus, we propose a scheme to generate a new data set using the real network data set which 1) has finer-grained time granularity and 2) considers the bursty behavior of traffic data. Simulation results show that using proposed methods, considerable energy saving is obtained, compared to the baselines at cost of negligible number of delayed users. Moreover, the proposed digital twin model can predict the performance of the DQN proactively in terms of RDM hence preventing the performance degradation in the network in anomalous situations.

Scientometric Insights of Digital Twins in Industry 4.0

“…The primary cited article within this cluster is authored by Kasi, Sk (2021.0-JAN) [41], published in the IEEE Internet of Things Journal. Notable members in this cluster include Italy (71), Spain (37), and France (33). Cluster #1, the second most significant, labeled Digital Twin, encompasses 14 members with a silhouette value of 0.854.…”

“…The major citing article in this cluster is authored by Huang, Z (2021.0-JAN) [38]. The most cited topics within this cluster include "artificial intelligence" (110), "manufacturing process" (109), and "machine learning" (71). The fifth largest cluster #4 includes 71 members with a silhouette value of 0.647 and is labeled as "energy efficiency" by LLR.…”

“…It is characterized by a neural network according to LLR. The primary citing article within this cluster is authored by Masoudi, M (2023.0-JAN) [71], pub-lished in IEEE Transactions on Vehicular Technology, Volume 72, Page 15, with DOI 10.1109/TVT.2022.3206498. The most frequently cited members in this cluster include 620 instances of digital twin, along with 289 instances each of deep learning and e-learning.…”

“…Prominent terms in this cluster include neural networks based on LLR. The primary citing article of this cluster is by Masoudi, M (2023.0-JAN) [71] published in the IEEE Transactions on Vehicular Technology. The most frequently cited members within this cluster are Digital Twin (876), Machine Learning (441), and E-Learning (424).…”

Dynamic Multi-Sleeping Control with Diverse Quality-of-Service Requirements in Sixth-Generation Networks Using Federated Learning

“…This method ensures that the AI-based sleep decision can only be used when the risk caused by entering base station sleep is less than the risk threshold set by the operator. If the risk exceeds the threshold, it indicates that the model has failed, the sleep decision should be suspended, and retraining should be carried out to ensure the reliability of the base station sleep mechanism [27]. The above studies focus only on the selection of base station sleep modes and do not consider the power control of the base stations, assuming that base stations use all their transmission power, without considering the impact of base station power control on coverage.…”