Discrete cosine transform (DCT) based orthogonal frequency division multiplexing (OFDM), which has double number of subcarrier compared to the classic discrete fourier transform (DFT) based OFDM (DFT-OFDM) at the same bandwidth, is a promising high spectral efficiency multicarrier techniques for future wireless communication. In this paper, an enhanced DCT-OFDM with index modulation (IM) (EDCT-OFDM-IM) is proposed to further exploit the benefits of the DCT-OFDM and IM techniques. To be more specific, a pre-filtering method based DCT-OFDM-IM transmitter is first designed and the non-linear maximum likelihood (ML) is developed for our EDCT-OFDM-IM system. Moreover, the average bit error probability (ABEP) of the proposed EDCT-OFDM-IM system is derived, which is confirmed by our simulation results. Both simulation and theoretical results are shown that the proposed EDCT-OFDM-IM system exhibits better bit error rate (BER) performance over the conventional DFT-OFDM-IM and DCT-OFDM-IM counterparts. Index Terms-Discrete cosine transform (DCT), index modulation (IM), performance analysis.