On Recurrent Neural Networks for Sequence-based Processing in Communications

Abstract

In this work, we analyze the capabilities and practical limitations of neural networks (NNs) for sequence-based signal processing which can be seen as an omnipresent property in almost any modern communication systems. In particular, we train multiple state-of-the-art recurrent neural network (RNN) structures to learn how to decode convolutional codes allowing a clear benchmarking with the corresponding maximum likelihood (ML) Viterbi decoder. We examine the decoding performance for various kinds of NN architectures, beginning with classical types like feedforward layers and gated recurrent unit (GRU)-layers, up to more recently introduced architectures such as temporal convolutional networks (TCNs) and differentiable neural computers (DNCs) with external memory. As a key limitation, it turns out that the training complexity increases exponentially with the length of the encoding memory ν and, thus, practically limits the achievable bit error rate (BER) performance. To overcome this limitation, we introduce a new training-method by gradually increasing the number of ones within the training sequences, i.e., we constrain the amount of possible training sequences in the beginning until first convergence. By consecutively adding more and more possible sequences to the training set, we finally achieve training success in cases that did not converge before via naive training. Further, we show that our network can learn to jointly detect and decode a quadrature phase shift keying (QPSK) modulated code with sub-optimal (anti-Gray) labeling in one-shot at a performance that would require iterations between demapper and decoder in classic detection schemes.

Deep Ensemble of Weighted Viterbi Decoders for Tail-Biting Convolutional Codes

“…This empirically shows the generalization of the novel method to longer lengths. The training process remains as simple as before, even as the length increases; There is no need to enforce a curriculum based ramp-up method for convergence as in [ 14 ].…”

“…Contemporary literature is split between two different model choices: model-free and model-based. Model-free works include those in [ 12 , 13 , 14 ], which leverage on state-of-the-art (SOTA) neural architectures with high neuronal capacity (i.e., ones that are able to implement many functions). On the other hand, under model-based approaches [ 15 , 16 , 17 , 18 ], a classical decoder is assigned learnable weights and trained to minimize a surrogate loss function.…”

On Training Neural Network Decoders of Rate Compatible Polar Codes via Transfer Learning

“…For CRC-polar codes, a neural BP algorithm was proposed by Doan et al [ 10 ]. In [ 17 ], Tandler et al proposed an ordering of data for efficient training of the decoder of convolutional codes. Unlike the problems of applying general deep learning, NND has an advantage that it is very easy to generate a training data set.…”

“…Previous studies [ 5 , 6 , 7 , 8 , 9 , 10 , 11 , 17 ] considered learning an NND for a specific code of fixed length and rate. However, wireless standards normally use a class of codes of multiple parameter sets, since a receiver is required to have either multiple decoders each of which is specialized to a code or a decoder which is flexible to decode many codes.…”

Deep Ensemble of Weighted Viterbi Decoders for Tail-Biting Convolutional Codes

“…Contemporary literature is split between two different model choices: model-free and model-based. Model-free works include [12][13][14], which leverage on state-of-the-art (SOTA) neural architectures with high neuronal capacity (i.e. ones that are able to implement many functions).…”