Background Advances in earth observation and machine learning techniques have created new options for forest monitoring, primarily because of the various possibilities that they provide for classifying forest cover and estimating aboveground biomass (AGB). Methods This study aimed to introduce a novel model that incorporates the atom search algorithm (ASO) and adaptive neuro-fuzzy inference system (ANFIS) into mangrove forest classification and AGB estimation. The Ca Mau coastal area was selected as a case study since it has been considered the most preserved mangrove forest area in Vietnam and is being investigated for the impacts of land-use change on forest quality. The model was trained and validated with a set of Sentinel-1A imagery with VH and VV polarizations, and multispectral information from the SPOT image. In addition, feature selection was also carried out to choose the optimal combination of predictor variables. The model performance was benchmarked against conventional methods, such as support vector regression, multilayer perceptron, random subspace, and random forest, by using statistical indicators, namely, root mean square error (RMSE), mean absolute error (MAE), and coefficient of determination (R 2). Results The results showed that all three indicators of the proposed model were statistically better than those from the benchmarked methods. Specifically, the hybrid model ended up at RMSE = 70.882, MAE = 55.458, R 2 = 0.577 for AGB estimation.