Semantic-Aware Collaborative Deep Reinforcement Learning Over Wireless Cellular Networks

Abstract

Collaborative deep reinforcement learning (CDRL) algorithms in which multiple agents can coordinate over a wireless network is a promising approach to enable future intelligent and autonomous systems that rely on real-time decision making in complex dynamic environments. Nonetheless, in practical scenarios, CDRL face many challenges due to heterogeneity of agents and their learning tasks, different environments, time constraints of the learning, and resource limitations of wireless networks. To address these challenges, in this paper, a novel semantic-aware CDRL method is proposed to enable a group of heterogeneous untrained agents with semantically-linked DRL tasks to collaborate efficiently across a resource-constrained wireless cellular network. To this end, a new heterogeneous federated DRL (HFDRL) algorithm is proposed to select the best subset of semantically relevant DRL agents for collaboration. The proposed approach then jointly optimizes the training loss and wireless bandwidth allocation for the cooperating selected agents in order to train each agent within the time limitation of its realtime task. Simulation results show the superior performance of the proposed algorithm compared to state-of-the-art baselines.

Semantic Communications for 6G Future Internet: Fundamentals, Applications, and Challenges

“…One category concentrates on a reliable and efficient transmission by reducing the length of bits flow, in fact, which merely accomplishes a source compression [68]. The other category usually applies to the scenarios with multiple intelligent agents, wherein the extracted semantic information is mainly used to facilitate multi-agent cooperation and is not directly linked to the task of the agents themselves [69]. The details about the challenges and techniques of SE and semantic noise can be found in Section V-A and Section V-C, respectively.…”

“…However, DRL still faces some challenges in practical applications, such as slow convergence [97], overfitting problems [98], and poor exploration in complex environments [99]. Collaborative DRL (CDRL) is treated as a promising solution to the above issues, wherein the agents can share their experiences and collaboratively learn the optimal policy for their task [69]. Whereas, as the agents have different environments, tasks, and action spaces, it is challenging to filter the helpful agents.…”

“…Whereas, as the agents have different environments, tasks, and action spaces, it is challenging to filter the helpful agents. In this sense, some works integrated the semantic awareness (i.e., the useful information about the relationship between tasks of the different agents) in the agent-selection scheme [69], [100] to choose an optimal subset of semantically-related DRL agents for collaboration.…”

Semantic Communication Meets Edge Intelligence

“…To this end, [12] proposes to consider both structural similarity and semantic relatedness when selecting the source agents. To obtain the semantic relatedness, the target agent is trained for a fixed number of steps under the policy of the source agent, and the average return value is taken as the semantic relatedness.…”

“…• DL-based SE for Task Similarity: Although the proposed semantic relatedness metric in [12] helps to improve the transmission performance, the extra training steps to obtain the return value will greatly reduce the system efficiency. Furthermore, it remains unclear how the number of training steps is determined, thereby limiting the scalability of this hand-crafted method.…”