Planting vegetation is one of the practical solutions for reducing sediment transfer rates. Increasing vegetation cover decreases environmental pollution and sediment transport rate (STR). Since sediments and vegetation interact complexly, predicting sediment transport rates is challenging. This study aims to predict sediment transport rate under vegetation cover using new and optimized versions of the group method of data handling (GMDH). Additionally, this study introduces a new ensemble model for predicting sediment transport rates. Model inputs include wave height, wave velocity, density cover, wave force, D50, the height of vegetation cover, and cover stem diameter. A standalone GMDH model and optimized GMDH models, including GMDH- honey badger algorithm (HBA), GMDH- rat swarm algorithm (RSOA), GMDH- sine cosine algorithm (SCA), and GMDH- particle swarm optimization (GMDH-PSO), were used to predict sediment transport rates. As the next step, the outputs of standalone and optimized GMDH were used to construct an ensemble model. The MAE of the ensemble model was 0.145 m3/s, while the MAEs of GMDH-HBA, GMDH-RSOA, GMDH-SCA, GMDH-PSOA, and GMDH in the testing level were 0.176 m3/s, 0.312 m3/s, 0.367 m3/s, 0.498 m3/s, and 0.612 m3/s, respectively. The Nash–Sutcliffe coefficient (NSE) of ensemble model, GMDH-HBA, GMDH-RSOA, GMDH-SCA, GMDH-PSOA, and GHMDH were 0.95 0.93, 0.89, 0.86, 0.82, and 0.76, respectively. Additionally, this study demonstrated that vegetation cover decreased sediment transport rate by 90%. The results indicated that the ensemble and GMDH-HBA models could accurately predict sediment transport rates. Based on the results of this study, sediment transport rate can be monitored using the IMM and GMDH-HBA. These results are useful for managing and planning water resources in large basins.