Abstract. The intersymbol interference caused by dispersion, chirp, and a vestigial sideband filter in intensity modulation and a direct detection single carrier system is analyzed theoretically and numerically. An iterative nonlinear intersymbol interference cancellation technique is proposed and experimentally demonstrated in a 40-Gbps 16-QAM Mach-Zehnder modulator-based vestigial sideband intensity modulation and direct detection halfcycle Nyquist-subcarrier modulation system over a 100-km uncompensated standard single-mode fiber transmission for the first time. The experimental results show that 2.2-dB receiver sensitivity improvement is obtained at the forward error correction limit by using the iterative technique.