Due to the light scattered by atmospheric aerosols, the amplitude image contrast is degraded and the depth measurement is greatly distorted for time-of-flight (ToF) imaging in fog. The problem limits ToF imaging to be applied in outdoor settings, such as autonomous driving. To improve the quality of the images captured by ToF cameras, we propose a polarization phasor imaging method for image recovery in foggy scenes. In this paper, optical polarimetric defogging is introduced into ToF phasor imaging, and the degree of polarization phasor is proposed to estimate the scattering component. A polarization phasor imaging model is established, aiming at separating the target component from the signal received by ToF cameras to recover the amplitude and depth information. The effectiveness of this method is confirmed by several experiments with artificial fog, and the experimental results demonstrate that the proposed method significantly improves the image quality, with robustness in different thicknesses of fog.