In fifth-generation (5G) fronthaul networks, the wavelength division multiplexing passive optical network (WDM-PON) is considered as an important tool to address the advanced transmission requirements. The nonlinear effects occur when multiple wavelengths are transmitted simultaneously in the same optical fiber. In this paper, we investigate the four-wave mixing (FWM) and the cross-phase modulation (XPM) effects by using non-return-to-zero (NRZ), 4-level pulse amplitude modulation (PAM4), and optical duobinary (ODB) modulation formats in WDM-PON-based 5G fronthaul networks. The results show that ODB performs better when it comes to limiting the FWM and XPM effects because of its precoded attribute. Compared to the unpredictable sensitivity penalty based on NRZ and PAM4, the FWM-induced sensitivity penalty using ODB is reduced to 2[Formula: see text]dB. Moreover, the XPM-induced sensitivity penalty is measured as 1[Formula: see text]dB by adopting ODB, which is almost 5[Formula: see text]dB better than NRZ and PAM4 modulation formats.