Optical transmission impairments are most critical reason to reduce both capacity and achievable transmission distance in optical communication system. Advanced modulation formats are important keys to reduce optical transmission impairments in digital signal processing (DSP) based coherent wavelength division multiplexed (WDM) system. These formats are more robust to non-linearity of optical fiber and results in increasing both spectral efficiency and bandwidth. Coherent detection technique increases sensitivity of receiver up to 30 dB in comparison of direct detection. Simulink seems to be the best cost-effective platform for analyzing mathematical modeling of optical coherent system. In this paper, mathematical model has been developed to analyze the optical transmission impairments by employing the advanced polarization multiplexed QPSK modulation in DSP based WDM coherent system. The simulation results have shown bit error rate (BER) improvement up to 10−6 and enhanced transmission distance to 150 km. These values have seen to be in close proximity with experimental results. The developed mathematical model seems to be more useful for extending the existing WDM optical system for longer distances with acceptable BER.

Now-a-days optical fiber system based upon coherent detection for improving power and spectral efficiency. In digital coherent optical system, Linear Time invariant (LTI) and non-linear time invariant (Non-LTI) impairment are main factors which limits the transmission capacity and throughput. Chromatic dispersion and polarization mode dispersion are related to linear time invariant impairments, whereas Kerr nonlinearities and laser noise are related to Non-LTI impairments. Development and combination of coherent detection, multi-level modulation format and digital signal processor make the optical fiber communication system more efficient. DSP provide the ease compromising solution for mitigation of transmission impairments. DSP can be implemented at both transmitter (pre-compensation) and receiver (post compensation)to overcome the existing transmission impairments. In time domain approach, filtering input date is obtained by multiplication process rather than convolution process. FDE, high latency and complex multiplication and additions are the disadvantages, caused by FFT and IFFT. For large data, FIR filter is most efficient in frequency domain using either Overlap Add or Overlap Save method. Hybrid model of FDE/TDE is to be used for LTI impairments. In this paper, a novel mitigation technique based upon Digital signal processor (DSP) is proposed for linear transmission impairments and neglecting nonlinear impairments. CD is compensated in frequency domain and PMD is compensated in time domain due its time varying nature and also discuss the Comparison of different adaptive algorithms.