Estimating surface elevation changes in mangrove forests requires a technique to filter the mangrove canopy and quantify the changes underneath. Hence, this study estimates surface elevation changes underneath the mangrove canopy through vegetation filtering and Difference of DEM (DoD) techniques using two epochs of unmanned aerial vehicle (UAV) data carried out during 2016 and 2017. A novel filtering algorithm named Surface estimation from Nearest Elevation and Repetitive Lowering (SNERL) is used to estimate the elevation height underneath the mangrove canopy. Consequently, DoD technique is used to quantify the elevation change rates at the ground surface, which comprise erosion, accretion, and sedimentation. The significant findings showed that region of interest (ROI) 5 experienced the highest volumetric accretion (surface raising) at 0.566 cm3. The most increased erosion (surface lowering) was identified at ROI 8 at −2.469 cm3. In contrast, for vertical change average rates, ROI 6 experienced the highest vertical accretion (surface raising) at 1.281 m. In comparison, the most increased vertical erosion (surface lowering) was spotted at ROI 3 at −0.568 m. The change detection map and the rates of surface elevation changes at Kilim River enabled authorities to understand the situation thoroughly and indicate the future situation, including its interaction with sea-level rise impacts.