Shack-Hartmann wavefront sensor (SHWFS) has been widely used in adaptive optics (AO) systems to detect phase distortion characteristics. In laser communication, target detection, vision optics and other application fields, the performance of SHWFS is affected by bright skylight and scattered light, which restricted the working ability of the AO system severely. Therefore, this paper proposes a new polarized SHWFS (p-SHWFS) based on the principle of polarization imaging, which utilize the difference of state of polarization (SoP) between signal and stray light to improve the image contrast. The p-SHWFS can be composed simply by a micro-lens array and a linear polarization camera. The camera uses four directional polarizing filters at 0°, 135°, 45° and 90° on every four pixels. Thus, the degree of linear polarization (DoLP) and angle of linear polarization (AoLP) for the incident wavefront can be analyzed, and the signal-to-background ratio (SBR) can be improved in some certain depending on the difference of SoP. In this paper, we introduce the basic principle of the p-SHWFS and validate the feasibility and accuracy improvement by numerical simulation and practical experiments. The experimental results show that the p-SHWFS can improve obviously the measurement accuracy under strong stray light when the difference of SoP exists. That may give us some initial reference to reduce the influence of stray light in laser communication, target detection, vision optics and other application fields.