Based on records of the bottom elevations of 511 ancient water wells from published archaeological reports, we reconstructed the paleo-groundwater levels (PGWL) in urban areas of Chengdu, Changsha, Nanjing, Suzhou, Suqian, Yancheng, Fuzhou, and Guangzhou cities in southern China. Our PGWL reconstruction shows that PGWL varied in two patterns. In the inland monsoon region (Chengdu and Changsha), there was a low PGWL in Jin (AD 266–420) and South Song (AD 1127–1279), and a high PGWL in Tang (AD 618–907) and Ming (AD 1368–1644). In the coastal region (Yancheng, Fuzhou, and Guangzhou), there was a low PGWL in Jin (AD 266–420) and Ming (AD 1368–1644) but a high PGWL in Tang (AD 618–907) and Song (AD 960–1279). Via cross-wavelet transform and wavelet transform coherence analyses, we found that monsoon and temperature significantly drove the PGWL fluctuations at the inter-centennial scale. East Asian Summer Monsoon-induced precipitation has continuously affected cities in the inland monsoon area represented by Chengdu and Changsha over the past 2,500 years. It has also intermittently affected Nanjing and Suzhou when EASM intensified. In parallel, temperature influenced the PGWL in coastal cities such as Yancheng, Fuzhou, and Guangzhou via the changes in the sea level. Also, the temperature affected the PGWL in relatively inland cities during climatic anomalies such as the Medieval Warm Period and Little Ice Age. This study demonstrates the value of archaeological records in learning how climatic factors influence the PGWL variation and its mechanism.