Interplay of frequency domain chromatic dispersion estimation algorithm and probabilistic shaping

Abstract

In this paper, we present an investigation into the interaction between probabilistic shaping (PS) techniques and frequency-domain dispersion estimation algorithms within coherent digital signal processing (DSP) chips for photonic integrated circuits (PIC). For the first time, we analyze how PS affects the performance of dispersion estimators, revealing that increased shaping factors lead to flatter slopes around the optimal value in the frequency domain autocorrelation function used as a cost metric, thus complicating accurate dispersion parameter estimation and potentially causing severe erroneous estimations. Through Monte Carlo simulations, we assess the impact of dispersion estimator errors on the mutual information achieved by different shaped PS-64QAM signals, demonstrating that under certain conditions, the performance degradation resulting from the incompatibility between PS and dispersion estimation can severely impair the shaping gain provided by the PS technology. Our study underscores the need for future research in coherent DSP to address the compatibility issues between PS and dispersion estimation algorithms, either by developing robust new dispersion estimation methods resilient to PS or refining the probability distribution models of PS to minimize the influence of signal variations on estimation accuracy. This work provides valuable insights for enhancing the adaptive capabilities of photonic chips across diverse service scenarios.