One of the practical and financial solutions to increase the efficiency of weirs is to modify the geometry of the plan and increase the length of the weir to a specific width. This increases the discharge coefficient (Cd) of the weir. In this study, a new weir referred to pseudo-cosine labyrinth weir (PCLW) was introduced. A hybrid machine learning LXGB algorithm was introduced to estimate the Cd of the PCLW. The LXGB is a combination of the linear population size reduction history-based adaptive differential evolution (LSHADE) and extreme gradient boosting (XGB) algorithm. Seven different input scenarios were presented to estimate the discharge coefficient of the PCLW weir. To train and test the proposed method, 132 data series, including geometric and hydraulic parameters from PCLW1 and PCLW2 models were used. The root mean square error (RMSE), relative root mean square error (RRMSE), and Nash–Sutcliffe model efficiency coefficient (NSE) indices were used to evaluate the proposed approach. The results showed that the input variables were the ratio of the radius to the weir height (R/W), the ratio of the length of the weir to the weir height (L/W), and the ratio of the hydraulic head to the weir height (H/W), with the average values of RMSE = 0.009, RRMSE = 0.010, and NSE = 0.977 provided better results in estimating the Cd of PCLW1 and PCLW2 models. The improvement compared to SAELM, ANFIS-FFA, GEP, and ANN in terms of R2 is 2.06%, 3.09%, 1.03%, and 5.15%. In general, intelligent hybrid approaches can be introduced as the most suitable method for estimating the Cd of PCLW weirs.