A Unified Multi-Task Semantic Communication System for Multimodal Data

Abstract

Task-oriented semantic communication has achieved significant performance gains. However, the model has to be updated once the task is changed or multiple models need to be stored for serving different tasks. To address this issue, we develop a unified deep learning enabled semantic communication system (U-DeepSC), where a unified end-to-end framework can serve many different tasks with multiple modalities. As the difficulty varies from different tasks, different numbers of neural network layers are required for various tasks. We develop a multi-exit architecture in U-DeepSC to provide early-exit results for relatively simple tasks. To reduce the transmission overhead, we design a unified codebook for feature representation for serving multiple tasks, in which only the indices of these task-specific features in the codebook are transmitted. Moreover, we propose a dimension-wise dynamic scheme that can adjust the number of transmitted indices for different tasks as the number of required features varies from task to task. Furthermore, our dynamic scheme can adaptively adjust the numbers of transmitted features under different channel conditions to optimize the transmission efficiency. According to simulation results, the proposed U-DeepSC achieves comparable performance to the task-oriented semantic communication system designed for a specific task but with significant reduction in both transmission overhead and model size.

Toward Natively Intelligent Semantic Communications and Networking

“…As compared to models built for only one activity, multitasking models are characterized by i) drastically reduced storage requirements due to the multitasking ability in exchanging model parameters; and ii) if various linked tasks share semantic information, it becomes more efficient to train the model for numerous tasks at once and increase its performance. For example, although the codebook in [69] is trained using a multi-modal approach, its usefulness in multidomain settings is severely constrained. This is mostly due to the fact that data from various modalities and activities have distinct distributions, resulting in substantial variation in the encoded properties.…”

Advances and Challenges in Semantic Communications: A Systematic Review

“…The authors in [62] presented a unified semantic encoding framework for both image and text transmitters using Transformer and introduced a new semantic decoder network that includes a query module and an information fusion module. To address the issue of updating models when tasks change or multiple models need to be stored, the authors in [63] proposed a unified DL-enabled semantic communication system (U-DeepSC) that can handle multiple tasks with various modalities. A multi-exit architecture in U-DeepSC provides early-exit results for simple tasks, and a unified codebook for feature representation reduces transmission overhead by transmitting only indices of task-specific features.…”

“…For image data transmission, the authors in [83] proposed a multi-layer semantic representation method and a collaborative reasoning mechanism for heterogeneous networks enabled by multi-layer SKB. Moreover, for multi-task requirements and multi-modal data sources, the authors in [63] proposed a cross-task shared SKB consisting of discrete semantic basis vectors and joint training with semantic-channel coding, which can reduce transmission overhead and model size while achieving comparable performance to task-specific semantic communication frameworks.…”

Transceiver Cooperative Learning-aided Semantic Communications Against Mismatched Background Knowledge Bases

“…Owing to advances in deep learning, in particular natural language processing (NLP) and computer vision, digging the semantic meaning of data for transmission becomes possible. In recent years, semantic communication systems learned on background knowledge bases (KBs) at transceivers have been developed for delivering text [3]- [5], image [6], [7], speech [8], as well as multimodal data [9]. In semantic communication systems, the transmitter uses a semantic encoding module to extract semantic information based on its own KB, and the receiver uses a semantic decoding module to recover the meaning of messages based on its own KB.…”

“…In semantic communication systems, the transmitter uses a semantic encoding module to extract semantic information based on its own KB, and the receiver uses a semantic decoding module to recover the meaning of messages based on its own KB. To make the transceivers have the same interpretation of the transmitted semantic data, existing works, e.g., [3]- [9], assume the transceivers have the same KBs. However, in practice, the KBs of the transmitter and receiver may be the same initially, they may become different due to the variations of the environment and/or the strength of the device's ability to acquire data.…”