Abstract-This paper investigates optical coherent systems based on polarization multiplexing and high-order modulations such as phase-shift keying (PSK) signals and quadrature amplitude modulations (QAM). It is shown that a simple linear receiver processing is sufficient to perfectly demultiplex the two transmitted streams and to perfectly compensate for group velocity dispersion (GVD) and polarization mode dispersion (PMD). In addition, in the presence of a strong phase noise of the lasers at the transmitter and receiver, a symbol-by-symbol detector with decision feedback is able to considerably improve the receiver robustness with a limited complexity increase. We will also discuss the channel estimation and the receiver adaptivity to time-varying channel conditions as well as the problem of the frequency acquisition and tracking. Finally, a new two-dimensional (polarization/time) differential encoding rule is proposed to overcome a polarization-ambiguity problem. In the numerical results, the receiver performance will be assessed versus the receiver complexity.Index Terms-Electrical equalization, group velocity dispersion (GVD), intersymbol interference (ISI), optical coherent transmission systems, phase-shift keying (PSK), polarization mode dispersion (PMD), quadrature amplitude modulation (QAM).