Nonlinearities in optical fibers are regarded as the most significant barriers that endanger the effectiveness of the optical transmission system and pose a threat to communication quality. Four-wave mixing (FWM) is one of the most important nonlinear effects that greatly reduces the wavelength-division multiplexing (WDM) system performance at high data rates over extended transmission distances. This research examines, and assesses, numerically, the behavior of a 4-channel, 40 Gbps WDM system under the effect of the FWM under various tuning parameters, including dispersion, input power, and wavelength spacing. The system model was built using OptiSystem software, and then three different modulation formats, namely, Non-return-to-zero-frequency shift keying, Return-to-zero frequency shift keying, and differential phase shift keying (DPSK) are used to assess the FWM power penalty. The results demonstrate that the FWM power penalty obtained with 1 nm wavelength separation in the DPSK method is dramatically reduced to −35 dBm. This study also demonstrates that when power variation is taken into consideration, the DPSK modulation scheme delivers a lower bit error rate in comparison to other modulation schemes.