Rapid urbanization has dramatically increased the demand for river sand, leading to soaring sand extraction rates that often exceed natural replenishment in many rivers globally. However, our understanding of the geomorphic and social-ecological impacts arising from Sand Mining (SM) remains limited, primarily due to insufficient data on sand extraction rates. Conventionally, bathymetry surveys and compilation of declared amounts have been used to quantify SM budgets, but they are often costly and laborious, or result in inaccurate quantification. Here, for the first time, we developed a Remote Sensing (RS)-based Deep Learning (DL) framework to map SM activities and budgets in the Vietnamese Mekong Delta (VMD), a global SM hotspot. We trained a near real-time object detection system to identify three boat classes in Sentinel-1 imagery: Barge with Crane (BC), Sand Transport Boat (STB), and other boats. Our DL model achieved a 96.1% Mean Average Precision (mAP) across all classes and 98.4% for the BC class, used in creating an SM boat density map at an Intersection over Union (IoU) threshold of 0.50. Applying this model to Sentinel-1, 256,647 boats were detected in the VMD between 2014-2022, of which 17.4% were BC. Subsequently, the annual SM budget was estimated by correlating it with a recent riverbed incision map. Our results showed that, between 2015-2022, about 366 Mm3 of sand has been extracted across the VMD. The annual budget has progressively increased from 34.92 Mm3 in 2015 to 53.25 Mm3 in 2022 (by 52%), with an annual increment of around 2.79 Mm3. At the provincial-scale, Dong Thap, An Giang, Vinh Long, Tien Giang, and Can Tho were the locations of intensive mining, accounting for 89.20% of the total extracted volume in the VMD. Finally, our estimated budgets were validated with previous research that yielded a correlation coefficient of 0.99 (percentage bias of 2.65%). The automatic DL framework developed in this study to quantify SM budgets has a high potential to be applied to other deltas worldwide also facing intensive SM.