The high-dimensional quantum key distribution (HD-QKD) encoded by orbital angular momentum (OAM) presents significant advantages in terms of information capacity. However, perturbations caused by free-space atmospheric turbulence introduce random fluctuations in OAM transmittance, impacting the performance of the system. Currently, there exists a gap in theoretical analysis concerning the statistical distribution of OAM-encoded QKD systems. In this study, we analyze the security of QKD systems by combining probability distribution of transmission coefficient (PDTC) of OAM with decoy-state BB84 method. To address the problem that the invalid key rate is calculated in the part transmittance interval of the post-processing process, an intelligent threshold method based on neural network is proposed to improve OAM-encoded QKD, which aims to conserve computing resources and enhance system efficiency. Our findings reveal that the ratio of root mean square (RMS) OAM-beam radius to Fried’s atmospheric coherence width plays a crucial role in ensuring secure key generation. Moreover, the training error of neural network is at the magnitude around 10 −3 , indicating the ability to predict optimization parameters quickly and accurately. This research contributes to the advancement of parameter optimization and prediction for free-space OAM-encoded HD-QKD systems. Furthermore, it provides valuable theoretical insights to support the development of free-space experimental setups.