The recent decade is an era of the 5G and more communication system technology. It has a wide range of application which requires wide bandwidth and high speed of bit rate. Also, Radio frequency and optical fiber is combined in order to attain high capacity transmission at lower costs which Radio over Fiber (RoF) is being as present trend of large broadband communication. This study investigates the RoF-Dense Wavelength Division Multiplexing (DWDM) system in order to enhance the capacity and performance of the system, and simulation was done by using optisystem software 14. The goal of the study was to enhance the number of channels and data transmission rate which the performance of the system was extremely reduced. Hence, different modulation techniques including Frequency Modulation (FM), Phase Modulation (PM), Frequency Shift Keying (FSK), and Continuous Phase Frequency Shift Keying (CPFSK) were presented for the DWDM-RoF system for various sample channels including ch1, ch8, ch16, ch24, and ch32 respectively. In addition, the performance of proposed DWDM-RoF system enhanced, which optimum value of Q factor and minimum BER was attained with FM modulation techniques as a comparison with other modulation techniques including PM, FSK, and CPFSK respectively.