In this study, an energy spectrum (ES) algorithm is proposed to retrieve wind direction from Xband marine radar image sequences. This algorithm is based on utilizing the occlusion area zero-pixel percentage (OZPP) to distinguish rain-free and rain-contaminated radar data. And then the rain-contaminated images are detected and discarded. The effect of radar radial attenuation in radar image sequences is modified by the piecewise fitting technique. Wind direction is determined from rain-free and radial correction data, based on the energy spectrum of small-scale wind streaks. The energy spectrum of small-scale wind streaks is obtained by establishing an energy spectrum scale separation filter. Based on the wind streak characteristics, a two-dimensional fast Fourier transform (FFT) is used to obtain the energy spectrum of radar images. The wind streak characteristics are derived from the distribution of the azimuth normalization radar cross-section (NRCS). The proposed algorithm is tested using data collected from X-band radar images and in-situ anemometer data from the coast of the East China Sea. Compared with the anemometer data, after using the proposed algorithm, the root-mean-square difference for wind direction is 12.13°, which is an acceptable result for engineering application.