Abstract-There are three major advantages for coherent optical orthogonal frequency-division multiplexing (CO-OFDM) transmission using digital signal processing. First, coherent detection is realized by digital phase estimation without the need for optical phase-locked loop. Second, OFDM modulation and demodulation are realized by the well-established computation-efficient fast Fourier transform (FFT) and inverse FFT. Third, adaptive data rates can be supported as different quadrature amplitude modulation (QAM) constellations are software-defined, without any hardware change in transmitter and receiver. However, it is well-known that coherent detection, OFDM, and QAM are all susceptible to phase noise. In this paper, theoretical, numerical, and experimental investigations are carried out for phase noise effects on high spectral efficiency CO-OFDM transmission. A transmission model in the presence of phase noise is presented. By using simulation, the bit error rate floors from finite laser linewidth are presented for CO-OFDM systems with high-order QAM constellations. In the experiments, the phase noise effects from both laser linewidth and nonlinear fiber transmission are investigated. The fiber nonlinearity mitigation based on receiver digital signal processing is also discussed.