SummaryIn this paper, we have proposed a high‐performance transmission system of 32 channels based on dense wavelength division multiplexing (DWDM) through an optical fiber link consisting of 110 km per span. To check the influence of nonlinear effects, we have considered phase modulation (SPM), cross‐phase modulation (XPM), four‐wave mixing (FWM), stimulated Raman scattering (SRS), and self‐steepening (SS) which limits the performance of optical communication systems. In the proposed system, each channel is modulated at 200 Gbps of dual‐polarization quadrature phase shift keying (DP‐QPSK) format, with a 50‐GHz grid yield whole data rate of 6.4 Tbps. The results have proved that the nonlinear phenomena degrade the performance of proposed optical fiber communication systems, and XPM has the most dominant effect, followed by SS, SRS, and SPM. The simulation was carried out with Optisystem 14 which is powerful software for modeling and simulation of optical fiber transmission.