This study introduces a new machine learning-based algorithm for the retrieving significant wave height (SWH) using synthetic aperture radar (SAR) images. This algorithm is based on the azimuthal cut-off wavelength and was developed in quad-polarized stripmap (QPS) mode in coastal waters. The collected images are collocated with a wave simulation from the numeric model, called WAVEWATCH-III (WW3), and the current speed from the HYbrid Coordinate Ocean Model (HYCOM). The sea surface wind is retrieved from the image at the vertical–vertical polarization channel, using the geophysical model function (GMF) CSARMOD-GF. The results of the algorithm were validated against the measurements obtained from the Haiyang-2B (HY-2B) scatterometer, yielding a root mean squared error (RMSE) of 1.99 m/s with a 0.82 correlation (COR) and 0.27 scatter index of wind speed. It was found that the SWH depends on the wind speed and azimuthal cut-off wavelength. However, the current speed has less of an influence on azimuthal cut-off wavelength. Following this rationale, four widely known machine learning methods were employed that take the SAR-derived azimuthal cut-off wavelength, wind speed, and radar incidence angle as inputs and then output the SWH. The validation result shows that the SAR-derived SWH by eXtreme Gradient Boosting (XGBoost) against the HY-2B altimeter products has a 0.34 m RMSE with a 0.97 COR and a 0.07 bias, which is better than the results obtained using an existing algorithm (i.e., a 1.10 m RMSE with a 0.77 COR and a 0.44 bias) and the other three machine learning methods (i.e., a >0.58 m RMSE with a <0.95 COR), i.e., convolutional neural networks (CNNs), Support Vector Regression (SVR) and the ridge regression model (RR). As a result, XGBoost is a highly efficient approach for GF-3 wave retrieval at the regular sea state.