The dense wavelength division multiplexing (DWDM) technique has been used to provide a large capacity and low bandwidth loss for optical communication systems. In this paper, simulation designs by Optisystem15.0 of carrier-suppressed return to zero (CSRZ), differential phase shift keying (DPSK), and intensity modulation is proposed to determine which modulation format is more compatible with four and eight DWDM channels for transmitting an optical signal over 400 km distance. For a long optical path, the dispersion compensation fiber (DCF) technique is proposed to eliminate dispersion effects and increase the possibility of transmitting multiple optical wavelengths over long single-mode fiber. Optical amplifiers are used to amplify the optical signal with a distorted signal and process the attenuation caused by the long transmission distance. In DCF network design, CSRZ offers the best performance because of the large quality factor (24.560) and high threshold power (15 dBm), which make the system compatible with increased distance between the transmitter and receiver; next is intensity modulation with a 24.5604 quality factor and 13 dBm threshold power value; DPSK comes in last with the worst performance, with a quality factor of 10 at 13 dBm power due to non-linear effects, especially non-linear phase noise. In the repeater design, the DPSK modulation format has the best performance with a large quality factor of 20.7913 at a high threshold power of 14 dBm for 150 GHz spacing; this is because the repeater technique is compatible with reducing the non-linear effects of the DPSK format. CSRZ and intensity modulation have the same performance with a 12 quality factor at 4 dBm power for intensity modulation and 3 dBm power for CSRZ modulation.