Refraction is an important factor influencing radiative transfer since it can modify the propagation trajectory and polarization states of lights; therefore, it is necessary to quantitively evaluate the effect of atmospheric refraction on radiative transfer process. To this end, a new atmospheric radiative transfer model including refraction process is proposed. The model accuracy is validated against benchmark results, literature results, and well‐tested radiative transfer models such as discrete coordinate method and RT3/PolRadtran. The impact of atmospheric refraction on both polarized radiance and fluxes is discussed for pure Rayleigh scattering atmosphere, atmosphere with aerosol, and cloud. The results show that atmospheric refraction has a significant influence on both the radiance and polarization states of diffuse light, where the relative change of the radiance of reflected light and transmitted light due to refraction can achieve 6.3% and 7.4% for Rayleigh scattering atmosphere, 7.2% and 7.8% for atmosphere with aerosol, and 6.2% and 6.8% for cloudy atmosphere, respectively. The relative change of the degree of polarization ranges from near zero in the horizon to 9.5% near neutral points. The angular distribution pattern of the relative change of the radiance for atmosphere with aerosol and cloud is very similar to that for pure Rayleigh scattering case, where its magnitude decreases gradually with the increasing of zenith angle for reflected light; but for transmitted light, the variation characteristics is opposite. The impact of refraction is gradually enhanced with the increasing of solar zenith angles and the optical depth of aerosol and cloud. As the wavelength of incident light increases, the impact declines rapidly for Rayleigh scattering medium. The relative change of the fluxes due to refraction is most notable for Middle Latitude Winter profile (about 8.2043% and 7.3225% for the transmitted and reflected light, respectively, at 0.35 µm). With increasing the optical depth of aerosol, the influence of refraction on the fluxes is gradually enhanced. For cloudy atmosphere, the relative changes of the fluxes due to refraction are not very sensitive to the variation of cloud optical depth and effective radius of cloud drops.