In this paper, we enhance the performance efficiency of the free-space optical (FSO) communication link using the hybrid on-off keying (OOK) modulation, M-ary digital pulse position modulation (M-ary DPPM), and M-pulse amplitude and position modulation (M-PAPM). This work analyzes and enhances the bit error rate (BER) performance of the moment generating function, modified Chernoff bound, and Gaussian approximation techniques. In the existence of both an amplified spontaneous emission (ASE) noise, atmospheric turbulence (AT) channels, and interchannel crosstalk (ICC), we propose a system model of the passive optical network (PON) wavelength division multiplexing (WDM) technique for a dense WDM (DWDM) based on the hybrid fiber FSO (HFFSO) link. We use eight wavelength channels that have been transmitted at a data rate of 2.5 Gbps over a turbulent HFFSO-DWDM system and PON-FSO optical fiber start from 1550 nm channel spacing in the C-band of 100 GHz. The results demonstrate (2.5 Gbps × 8 channels) 20 Gbit/s-4000 m transmission with favorable performance. In this design, M-ary DPPM-M-PAPM modulation is used to provide extra information bits to increase performance. We also propose to incorporate adaptive optics to mitigate the AT effect and improve the modulation efficiency. We investigate the impact of the turbulence effect on the proposed system performance based on OOK-M ary- PAPM-DPPM modulation as a function of M-ary DPPM-PAPM and other atmospheric parameters. The proposed M-ary hybrid DPPM-M-PAPM solution increases the receiver sensitivity compared to OOK, improves the reliability and achieves a lower power penalty of 0.2–3.0 dB at low coding level (M) 2 in the WDM-FSO systems for the weak turbulence. The OOK/M-ary hybrid DPPM-M-PAPM provides an optical signal-to-noise ratio of about 4–8 dB of the DWDM-HFFSO link for the strong turbulence at a target BER of 10−12. The numerical results indicate that the proposed design can be enhanced with the hybrid OOK/M-DPPM and M-PAPM for DWDM-HFFSO systems. The calculation results show that PAPM-DPPM has increased about 10–11 dB at BER of 10−12 more than the OOK-NRZ approach. The simulation results show that the proposed hybrid optical modulation technique can be used in the DWDM-FSO hybrid links for optical-wireless and fiber-optic communication systems, significantly increasing their efficiency. Finally, the use of the hybrid OOK/M-ary DPPM-M-PAPM modulation schemes is a new technique to reduce the AT, ICC, ASE noise for the DWDM-FSO optical fiber communication systems.