Advanced modulation format is a promising solution to accomplish improved bandwidth exploitation for quality transmission in optical networks. This paper proposes a modified duobinary modulation in an optical communication network, and the proposed system is compared with the preceding modulation scheme of duobinary modulation without a precoder and duobinary modulation with a precoder. Ideally, two or more signals are transmitted over single-mode fiber using a multiplexing method. Hence, wavelength division multiplexing (WDM) using an erbium-doped fiber amplifier (EDFA) as an active optical network is utilized to improve the quality factor and reduce the effect of intersymbol interference in optical networks. The performance of the proposed system is analyzed for parameters such as quality factor, bit error rate, and extinction ratio using OptiSystem 14 software.