As the demand for higher data throughput in coherent optical communication systems increases, we need to find ways to increase capacity in existing and future optical communication links. To address the demand for higher spectral efficiencies, we apply end-to-end optimization for joint geometric and probabilistic constellation shaping in the presence of Wiener phase noise and carrier phase estimation. Our approach follows state-of-the-art bitwise auto-encoders, which require a differentiable implementation of all operations between transmitter and receiver, including the DSP algorithms. In this work, we show how to modify the ubiquitous blind phase search (BPS) algorithm, a popular carrier phase estimation algorithm, to make it differentiable and include it in the end-to-end constellation shaping. By leveraging joint geometric and probabilistic constellation shaping, we are able to obtain a robust and pilotfree modulation scheme improving the performance of 64-ary communication systems by at least 0.1 bit/symbol compared to square QAM constellations with neural demappers and by 0.05 bit/symbol compared to previously presented approaches applying only geometric constellation shaping.

The performance of a wideband coherent receiver was investigated. The relative impact of digital pre-distortion, geometric constellation shaping and pilot sequence detection, as well as the number of sub-channels in the super-channel, on the receiver performance was explored. The detection of a net data rate of 2.36 Tb/s after 75 km transmission of a 8 × 26 GBd DP-GS-256-QAM super-channel was demonstrated using a single 110 GHz electrical bandwidth receiver. The overall improvement due to the digital pre-distortion and tailored geometric constellation shaping was 1.2 bit/4D-sym in the achievable information rate.