Task-Oriented Multi-User Semantic Communications

Abstract

While semantic communications have shown the potential in the case of single-modal single-users, its applications to the multi-user scenario remain limited. In this paper, we investigate deep learning (DL) based multi-user semantic communication systems for transmitting single-modal data and multimodal data, respectively. We will adopt three intelligent tasks, including, image retrieval, machine translation, and visual question answering (VQA) as the transmission goal of semantic communication systems. We will then propose a Transformer based unique framework to unify the structure of transmitters for different tasks. For the single-modal multi-user system, we will propose two Transformer based models, named, DeepSC-IR and DeepSC-MT, to perform image retrieval and machine translation, respectively. In this case, DeepSC-IR is trained to optimize the distance in embedding space between images and DeepSC-MT is trained to minimize the semantic errors by recovering the semantic meaning of sentences. For the multimodal multi-user system, we develop a Transformer enabled model, named, DeepSC-VQA, for the VQA task by extracting text-image information at the transmitters and fusing it at the receiver. In particular, a novel layer-wise Transformer is designed to help fuse multimodal data by adding connection between each of the encoder and decoder layers. Numerical results will show that the proposed models are superior to traditional communications in terms of the robustness to channels, computational complexity, transmission delay, and the task-execution performance at various task-specific metrics.

Semantic Communications: Principles and Challenges

“…Different from all the aforementioned works for the point-to-point transmission, MU-DeepSC is designed for serving multi-user transmission. As an extension of DeepSC, A Transformer based framework [72] has been developed as a unique structure for serving different tasks. Various tasks have been tested…”

Adaptable Semantic Compression and Resource Allocation for Task-Oriented Communications

“…The authors in [16] proposed an E2E learningdriven architecture of semantic communication to integrate the semantic inference and physical layer communication problems, where the transceiver is optimized jointly to reach Nash equilibrium while minimizing the average semantic errors. Recently, deep learning (DL) has emerged as a popular solution for semantic communications due to its powerful feature extraction capability [17]- [26]. Farsad et al [17] developed a long short-term memory (LSTM) enabled joint source-channel coding (JSCC) for the transmission of text data.…”

“…For multimodal data transmission, Xie et al [25] developed MU-DeepSC for the visual question answering task, where one user transmits text-based questions about images, and the inquiry images are transmitted from another user. Based on MU-DeepSC, a Transformer based framework [26] has been developed as a unique structure for serving different tasks. Various tasks have been tested in [26] to show its superiority.…”

“…Based on MU-DeepSC, a Transformer based framework [26] has been developed as a unique structure for serving different tasks. Various tasks have been tested in [26] to show its superiority. In summary, the existing works on semantic communications are focused on the implementation of semantic communication systems, in which extracted semantics are compressed via a fixed neural network or directly transmitted without further compression.…”

Semantic-aware Speech to Text Transmission with Redundancy Removal

“…A deep learning-based semantic communication system has been developed for efficient and robust transmission of text in [2], the deep learning model of which is then further compressed to be able to work on IoT devices [8]. [9] and [10] designed semantic communication systems that are capable of multimodal data transmission for tasks, such as visual question answering.…”