Deepcode: Feedback Codes via Deep Learning

Abstract

The design of codes for communicating reliably over a statistically well defined channel is an important endeavor involving deep mathematical research and wide-ranging practical applications. In this work, we present the first family of codes obtained via deep learning, which significantly beats state-of-the-art codes designed over several decades of research. The communication channel under consideration is the Gaussian noise channel with feedback, whose study was initiated by Shannon; feedback is known theoretically to improve reliability of communication, but no practical codes that do so have ever been successfully constructed.We break this logjam by integrating information theoretic insights harmoniously with recurrent-neural-network based encoders and decoders to create novel codes that outperform known codes by 3 orders of magnitude in reliability. We also demonstrate several desirable properties of the codes: (a) generalization to larger block lengths, (b) composability with known codes, (c) adaptation to practical constraints. This result also has broader ramifications for coding theory: even when the channel has a clear mathematical model, deep learning methodologies, when combined with channel-specific information-theoretic insights, can potentially beat state-of-the-art codes constructed over decades of mathematical research.

DeepJSCC-f: Deep Joint Source-Channel Coding of Images With Feedback

“…Although we have considered Deepcode as channel code for our experiments, we found that the proposed scheme cannot be easily employed for the transmission of images in large blocklengths. The results presented in [30] consider transmission of a message of 50 bits at rate 1/3 on the forward channel SNR range -2dB -1dB. In this range (-2dB -1dB), indeed Deepcode outperforms other channel codes; however, we found that none of our source codes could achieve a non-trivial PSNR value that improves upon the quality obtained when averaging all pixels with their average values (lower bound considered in case of failed transmissions).…”

“…While it is shown in [30] that the proposed DNN-based feedback channel code provides some level of robustness against noise in the feedback channel, to the best of our knowledge, this is the first practical JSCC scheme that is robust to feedback channel noise. Figure 12 shows the performance for different feedback channel SNRs (0 dB, 10 dB, 20 dB, and noiseless) and the single layer model (no feedback).…”

“…Hyeji et al [30] present a practical DNN-based channel code, Deepcode, that exploit the feedback channel, outperforming other codes such as LDPC and Turbo code. Although we have considered Deepcode as channel code for our experiments, we found that the proposed scheme cannot be easily employed for the transmission of images in large blocklengths.…”

“…More impressive results have been achieved through DNN-based designs in settings where current codes fall short of the fundamental theoretical limits; for example, for channels that are harder to model, such as optical [28] and molecular [29] communications, or in settings no practical coding scheme has been known even for known channel statistics, such as communication in the presence of channel output feedback [30]. In [30], authors constructed a code based on recurrent neural networks (RNNs), called Deepcode, that can significantly outperform the coding scheme of Schalkwijk and Kailath [19] in the case of noiseless feedback, and achieve reasonable reliability even in the case of noisy channel output feedback. Although the authors proposed concatenating Deepcode with other known coding schemes to achieve larger blocklengths, the code itself is still limited to short blocklengths due to training difficulty.…”

Autoencoder-Based Error Correction Coding for One-Bit Quantization

“…Most of the prior studies are aimed at learning and/or improving the performance of decoding algorithms through the use of a neural network. There are only a few papers that aim to learn an encoder, which is more difficult than learning a decoder due to the difficulties of training the lower layers in deep networks [13], [14], [15]. We specifically design a channel code for the challenging one-bit quantized AWGN channels via an autoencoder to obtain reliable communication at the Shannon rate.…”