In this paper, it is proposed to enhance coherent optical orthogonal frequency-division multiplexing (OFDM) system by using index modulation in terms of bit error rate (BER) and spectral efficiency for different levels of the fiber nonlinearity. In the coherent optical OFDM with index modulation system (CO-OFDM-IM), the information is conveyed not only through activated subcarriers but also by indices of the active subcarrier that are dynamically updated at the transmitted symbol rates. Unlike the conventional coherent optical OFDM (CO-OFDM) system, adjusting the number of the active subcarriers in the CO-OFDM-IM system provides improved BER performance and reduction the signal peak-to-averageratios, when the fiber nonlinearity becomes significant. The simulation results illustrate that the proposed CO-OFDM-IM system has better BER performance when the proposed CO-OFDM-IM system and the CO-OFDM system have similar spectral efficiency. Also, the CO-OFDM-IM system can provide better spectral efficiency than the CO-OFDM system at the low-to-medium order modulation such as BPSK, QPSK, and 16QAM.INDEX TERMS Coherent optical communications, OFDM, OFDM with index modulation, spectral efficiency.