The main work of this paper is to explore the influence of swell wave on retrieval of wind speed using ENVISAT ASAR wave mode imagery. The normalized radar cross section (NRCS) scene under different sea states is simulated to investigate the relationship between NRCS variation with swell height, together with swell direction. Moreover, the key parameter of imagery variance (Cvar) is selected to describe the swell wave on SAR imagery. In addition, the imagery parameters of skewness and kurtosis are together analyzed as a function of collocated significant swell wave height and wind speed. Based on the analyzed results, a new method for wind speed retrieval is proposed using ENVISAT ASAR, namely, F(n). Besides the CMOD parameters of NRCS, incidence angle, and relative wind direction, the imagery parameters of Cvar, skewness, and kurtosis are used to compensate for the influence of swell wave on wind speed retrieval in F(n). Finally, the collocated European Centre for Medium-Range Weather Forecast (ECMWF) wind speed dataset and ENVISAT ASAR wave mode imagery are used to verify the retrieval precision and compare with CMOD functions. It is concluded that the F(n) model performs much better than other CMOD functions, with a correlation of 0.89, a bias of 0.08, a RMSE of 1.2 m/s, and an SI of 0.1.