An essential feature of a long-haul Optical Communication System (OCS) is to transmit huge volumes of data over long distances. In OCS, fiber dispersion and Four-Wave Mixing (FWM) play negative roles in achieving this, as the propagation of the light signal, associated with the pulse, travels at varying speeds. Dense Wavelength Division Multiplexing (DWDM) system is not spared from these identified limitations. In this work, a mathematical analysis for the dispersion and FWM was presented and validated through simulation software. The research was designed and implemented using an optical simulation software tool, OptiSystem 17.0. Transmission parameters of dispersion and Nonlinear (NL) effects due to refractive index, FWM, were considered. The research was carried out on a 125 km-long fiber link and was investigated under a 16-channel DWDM system. The effects of identified limitations on the performance of the DWDM system with channel arrangement of 100, 50, and 25 GHz were further explored. This work compared the DWDM system under these different channel arrangements. The transmission performance of the DWDM system was evaluated based on signal power, noise power, and eye pattern. The results obtained show that any increase in capacity causes significant decreases in the performance of the DWDM system. It was, however, noted that noise power is independent of an increase in bit rate. The results obtained further revealled that the launched power of the system must be reduced for better performance of the DWDM system and improved immunity against identified limitations.