Free space optical (FSO) communication offers huge bandwidth, license-free spectrum and a more secure channel. PIN diodes are normally used for detection, but avalanche photodiodes (APD) are preferred for detecting high-speed FSO signals in many applications. In the case of APD, the noise distribution is input-dependent Gaussian noise (IDGN) rather than input-independent Gaussian noise (IIGN). We investigate the error analysis using on-off keying (OOK) for various detection approaches. This paper proposes a machine learning approach and compares its performance with soft and hard decisions. Soft values in the case of IDGN and IIGN are derived, and the optimum and sub-optimum detection thresholds are evaluated. The proposed novel ML approach shows better performance gains than the other approaches. It is also demonstrated that the IDGN model should have an optimum detection and achieve a gain of 2.5[dB] and about 1[dB] at λ = 0[dB] and λ = 10[dB], respectively. Experimental results are plotted for the FSO channel data, and a model fit curve is plotted using the ML approach.INDEX TERMS Error analysis, machine learning, optical communications, hard decision, soft decision.