To suppress the influence of rainfall when extracting sea surface wind and wave parameters using X-band marine radar and control the quality of the collected radar image, it is necessary to detect whether the radar image is contaminated by rainfall. Since the detection accuracy of the statistical characteristics methods (e.g., the zero pixel percentage method and the high-clutter direction method) is limited and the threshold is difficult to determine, the machine learning methods (e.g., the support vector machine-based method and the neural network algorithm) are difficult to select appropriate quality and quantity of data for model training. Therefore, based on the feature that rainfall can change the sea surface texture, a wave texture difference method for rainfall detection is proposed in this paper. Considering the spatial rainfall is uneven, the polar coordinates of the radar image are converted into Cartesian coordinates to detect rainfall. To express the maximum wave difference more accurately, the calculation method of the pixels in the radar texture difference map is redefined. Then, a consecutive pixel method is used to detect the calculated radar texture difference map, and this method can detect adaptively with the change of wind. The data collected from the shore of Haitan Island along the East China Sea are used to validate the effectiveness of the proposed method. Compared with the zero pixel percentage method and the support vector machine-based method, the experimental results demonstrate that the proposed method has better rainfall detection performance. In addition, the research on the applicability of the proposed method shows that the wave texture difference method can finish the task of rainfall detection in most marine environments.