Quantification of shading effects from complex terrain on solar radiation is essential to obtain precise data on incident solar radiation in mountainous areas. In this study, a machine learning (ML) approach is proposed to rapidly estimate the shading effects of complex terrain on solar radiation. Based on two different ML algorithms, namely, Ordinary Least Squares (OLS) and Gradient Boosting Decision Tree (GBDT), this approach uses terrain-related factors as input variables to model and analyze direct and diffuse solar radiation shading rates. In a case study of western Sichuan, the annual direct and diffuse radiation shading rates were most correlated with the average terrain shading angle within the solar azimuth range, with Pearson correlation coefficients of 0.901 and 0.97. The GBDT-based models achieved higher accuracy in predicting direct and diffuse radiation shading rates, with R2 values of 0.982 and 0.989, respectively, surpassing the OLS-based models by 0.081 and 0.023. In comparisons between ML models and classic curve-fitting models, the GBDT-based models consistently performed better in predicting both the direct radiation shading rate and the diffuse radiation shading rate, with a standard deviation of residuals of 0.330% and 0.336%. The OLS-based models also showed better performance compared to the curve-fitting models.