Baise city is one of the largest producers of mangoes, with this agricultural industry located in the karst region of Southwest China. However, calcium-rich and alkaline soils, severe soil fragmentation, and poor water and fertilizer retention capacity contribute to low mango yields and are key issues that limit the development of the mango industry in karst areas. Our study objectives were to identify the soil factors that limit mango growth and yield in the karst region of Southwest China, and to determine how these growth- and production-limiting conditions vary between landscape positions. This study analyzed the differences in soil nutrient and element contents in mango leaves, and used a Random Forest algorithm to calculate the eigenvalues of the mango leaf and soil elemental indices in the different geomorphic parts (slopes, transition zone, passes, high-yielding depressions, and low-yielding depressions) of the karst peak-cluster depressions. The key factors affecting the mango leaves and soil were screened based on the diagnostic results and the eigenvalues. The results showed that for the elemental contents of Fe, Mg, Ca, and Mn in the mango leaves in the different geomorphic parts of the karst, the peak-cluster depressions were generally deficient and varied significantly. The contents of available B (AB), soil organic matter (SOM), pH, total nitrogen, available Fe, available Mn, alkaline hydrolysis nitrogen, exchangeable Ca (Caex), exchangeable Mg, and other indices in the soils differed significantly, and AB, available Zn, and available K (AK) showed low or very low content levels. In addition, the key soil factors limiting mango yield in the karst areas were AB, fulvic acid, SOM, Fe, Mn, Caex, soil water, and AK; and the key mango leaf factors were Ca, Mn, Fe, Zn, and Mg. Consequently, the characteristics of soil water content, pH, and soil organic matter may be the main drivers affecting the differences in the mango yield and the elemental characteristics. These findings suggest that the addition of organic fertilizer could improve the quality and yield of mangoes in karst areas.