This paper presents a sky polarized orientation method based on the symmetry of the atmospheric polarization mode and the inertial sensors. Firstly, a method of extracting solar meridians based on canny operator and Hough Transform is proposed by using the symmetry of atmospheric polarization pattern. Then using the horizontal Angle information of the inertial sensor and the ephemeris information of the sun, a method for solving the yaw Angle of the polarized light is designed. Finally, a single axis rotation experiment and a vehicle-mounted dynamic experiment are carried out to verify and compare the proposed method. Compared with the least square method, the positioning accuracy of the single-axis turntable experiment is improved by 22.61%~73.86%, and that of the vehicle-mounted dynamic experiment is improved by 47.71%, which shows that the method can provide a new orientation means for unmanned vehicles and unmanned aerial vehicles.