Performance Versus Complexity Study of Neural Network Equalizers in Coherent Optical Systems

Abstract

We present the results of the comparative performance-versus-complexity analysis for the several types of artificial neural networks (NNs) used for nonlinear channel equalization in coherent optical communication systems. The comparison is carried out using an experimental set-up with the transmission dominated by the Kerr nonlinearity and component imperfections. For the first time, we investigate the application to the channel equalization of the convolution layer (CNN) in combination with a bidirectional long short-term memory (biLSTM) layer and the design combining CNN with a multilayer perceptron. Their performance is compared with the one delivered by the previously proposed NN-based equalizers: one biLSTM layer, three-dense-layer perceptron, and the echo state network. Importantly, all architectures have been initially optimized by a Bayesian optimizer. First, we present the general expressions for the computational complexity associated with each NN type; these are given in terms of real multiplications per symbol. We demonstrate that in the experimental system considered, the convolutional layer coupled with the biLSTM (CNN+biLSTM) provides the largest Q-factor improvement compared to the reference linear chromatic dispersion compensation (2.9 dB improvement). Then, we examine the trade-off between the computational complexity and performance of all equalizers and demonstrate that the CNN+biLSTM is the best option when the computational complexity is not constrained, while when we restrict the complexity to some lower levels, the three-layer perceptron provides the best performance.

Experimental Implementation of a Neural Network Optical Channel Equalizer in Restricted Hardware Using Pruning and Quantization

“…The hyperparameters that define the structure of the NN are obtained using a Bayesian optimizer 10,30 , where the optimization is carried out concerning the signal's restoration quality performance, similar to Ref. 10 . The resulting optimized MLP has three hidden layers (we did not optimize the number of layers, but only the number of neurons and the activation type), with 500, 10, and 500 neurons, respectively.…”

“…It should be stressed that in the telecommunication industry, the competition between possible solutions occurs not only in terms of performance but also in terms of hardware deployment options, operational costs, and power consumption. During the last years, the approaches based on machine learning techniques and, in particular, those utilizing NNs, have become an increasingly popular topic of research, as they can efficiently unroll both fiber and component-induced impairments [5][6][7][8][9][10][11][12][13][14][15] .…”

“…One of the straightforward ways of using a NN for signal's corruption compensation in optical transmission systems is to plug it into the system as a post-equalizer 7,10,14 , a special signal processing device at the receiver side, aimed at counteracting the detrimental effects emerging during the data transmission 16 . A number of NNs architectures have been already studied in different types of optical systems (submarine, long-haul, metro, and access).…”

“…A number of NNs architectures have been already studied in different types of optical systems (submarine, long-haul, metro, and access). These architectures include the feed-forward designs such as the MLP 7,10,14,15 , considered in the current study, or more sophisticated recurrent-type NN structures [10][11][12]17 . The predominant number of preceding studies have demonstrated the potential of using the NNs in the optical channel equalization task, especially in terms of the transmission quality improvement 7,8 .…”

“…However, the practical deployment of real-time NN-based channel equalizers implies that their computational complexity is, at least, comparable, or, desirably lower than that of existing conventional digital signal processing (DSP) solutions 18 , and stays the matter of debate. This is a critical bottleneck when deploying these algorithms on modern optical transceivers, because, typically, the good NN performance is linked to the use of a large number of parameters and floating-point operations 10 . A high computational complexity leads to high memory and computing power requirements, increasing the energy and resource consumption 19,20 .…”