Agricultural drought disaster is a major natural disaster affecting economic and social development. It is of significance to investigate the spatial–temporal pattern and the dominant influence of natural and human factors on agricultural drought disasters for drought hazard relief. In this study, Mann–Kendall test was adopted to explore the evolution of agricultural drought disasters. Random forest algorithm, which integrates feature importance and accumulated local effects plot, was applied to quantify the effect of principal influencing factors on agricultural drought disasters. Results show that over the period from 1950 to 2019, agricultural drought disasters in China have undergone significant fluctuations. The spatial pattern of agricultural drought disaster tends to decrease in severity from north to south. The total sown area of crops, precipitation, effective irrigation area, domestic patent application authorization, and regional GDP are the top 5 dominant factors influencing agricultural drought disasters. It also found that agricultural drought disaster negatively correlates with precipitation, domestic patent application authorization, and regional GDP, and the nonlinear response of agricultural drought disaster to total sown area of crops and effective irrigation area can be basically divided into two stages. In the first stage, with the increase of feature value, agricultural drought disaster is also increasing. In the second stage, with the increase of feature value, agricultural drought disaster is growing slow or just decreasing. The results can deepen the understanding of agricultural drought disasters and provide scientific basis for drought event monitoring, evaluation, and early warning.