Puding County in central Guizhou is a typical karst ecologically vulnerable area integrating rural, mountainous, and ecological migration areas. It is essential to study the dynamic trajectory and direction of its agroecosystems (AESs) transformation to promote sustainable land use development in karst mountain areas. Based on high-resolution remote sensing images from 2004 to 2020, this study analyzes the transformation characteristics, typical landscape patterns, and their driving forces of AESs using the landscape pattern index, moving window method, and Geodetector model. The results show that: (i) The overall landscape pattern of AESs in the study area developed towards complexity and diversification from 2004 to 2020. The overall trend of woodland and grassland (WGL) is increasing, the slope cultivated land (SCL) is shrinking, the construction land (CL) is spreading and expanding along the vicinity of the town, and the economic and fruit forests (EFF) are increasing significantly. (ii) Three typical patterns are concluded according to the main transformation directions of AESs: WGL restoration type, CL growth type, and EFF growth type; middle and high mountains are dominated by the WGL restoration type with increased ecological functions, and the concentration of the new WGL increases with the increase in elevation; low mountain areas are dominated by the CL growth type with enhanced living functions, and the concentration of the new CL decreases with the increase in slope and elevation; valleys and hills are dominated by the EFF growth type with enhanced ecological and economic functions, and the concentration of the new EFF increases with the decrease in slope and elevation. (iii) Socio-economic factors are the dominant drivers of AES transformation. The WGL restoration type is dominated by slope and elevation, the CL growth type is mainly influenced by slope and urbanization rate, and the EFF growth type is primarily influenced by soil organic matter, slope, urbanization rate, and resident population. The study results have implications for rural land use, rural development, and ecological restoration of rocky desertification in karst mountain areas and other similar mountainous areas.