A systematic understanding of the spatial-temporal evolution patterns of cultivated land fragmentation (CLF), its driving factors, and its relationship with the expansion of urban and rural construction land is essential for identifying strategies to mitigate CLF in rapidly urbanizing regions. This study combined landscape fragmentation with ownership fragmentation, analyzing CLF through three dimensions: resource endowment, spatial concentration, and convenience of utilization, with eight selected indicators. By comparing village-level data from 2013 to 2022, we explored the key drivers of CLF and its conflicts with urban and rural construction land expansion. The findings indicate a clear spatial variation in village-level CLF in Ezhou, characterized by low fragmentation in the northwest and northeast, and high fragmentation in the southwest and central regions. This pattern is in contrast to Ezhou’s economic development, which decreased progressively from east to north and south. Over the study period, village-level CLF in Ezhou evolved from being primarily moderately and relatively severely fragmented to predominantly severely and relatively severely fragmented, with an overall declining trend and more pronounced polarization. At the same time, the CLF within the village region demonstrated notable spatial clustering features, with a rapid increase observed between 2013 and 2022. It was also discovered that CLF is driven by various factors, with the main influences being the proportion of construction land, land use intensity, and population density. Cultivated land is the main source of both urban construction land (UCL) and rural construction land (RCL), with average contribution rates of 46.47% and 62.62%, respectively. This research offers empirical evidence for rapid urbanization and serves as a critical reference for rural revitalization and coordinated urban–rural development, with potential guidance for future policy formulation and implementation.