Accurate mapping of tea plantations is crucial for agricultural management and economic planning, yet it poses a significant challenge due to the complex and variable nature of tea cultivation landscapes. This study presents a high-precision approach to mapping tea plantations in Anji County, Zhejiang Province, China, utilizing multi-source remote sensing data and advanced deep learning models. We employed a combination of Sentinel-2 optical imagery, Sentinel-1 synthetic aperture radar imagery, and digital elevation models to capture the rich spatial, spectral, and temporal characteristics of tea plantations. Three deep learning models, namely U-Net, SE-UNet, and Swin-UNet, were constructed and trained for the semantic segmentation of tea plantations. Cross-validation and point-based accuracy assessment methods were used to evaluate the performance of the models. The results demonstrated that the Swin-UNet model, a transformer-based approach capturing long-range dependencies and global context for superior feature extraction, outperformed the others, achieving an overall accuracy of 0.993 and an F1-score of 0.977 when using multi-temporal Sentinel-2 data. The integration of Sentinel-1 data with optical data slightly improved the classification accuracy, particularly in areas affected by cloud cover, highlighting the complementary nature of Sentinel-1 imagery for all-weather monitoring. The study also analyzed the influence of terrain factors, such as elevation, slope, and aspect, on the accuracy of tea plantation mapping. It was found that tea plantations at higher altitudes or on north-facing slopes exhibited higher classification accuracy, and that accuracy improves with increasing slope, likely due to simpler land cover types and tea’s preference for shade. The findings of this research not only provide valuable insights into the precision mapping of tea plantations but also contribute to the broader application of deep learning in remote sensing for agricultural monitoring.