In order to achieve the fine alignment of strapdown inertial navigation (SINS) under large misalignment angles, a novel filtering alignment method is proposed based on the second-order extended Kalman filter (EKF2) and adaptive fuzzy inference system (AFIS). Firstly, the quaternion is employed to represent the attitude errors of SINS. A second-order nonlinear state equation is made based on the nonlinear velocity error model and attitude error model, and the linear measurement equation is based on the velocity outputs from SINS. Then, the filtering scheme is designed based on EKF2 and AFIS. The error estimation and fine alignment can be achieved by using the proposed filtering scheme. The results of Monte Carlo Simulation show that the errors of pitch, roll and yaw misalignment angles quickly decrease to about 14″, 15″ and 7.62′ respectively in 350 s under the condition of any misalignment angles with pitch error from −40° to 40°, roll error from −40° to 40°, and yaw error from −50° to 50°. Even when the initial misalignment angles are all very large such as (80°, 120°, 170°), the proposed nonlinear alignment method still can converge normally by utilizing the adaptive fuzzy inference system (AFIS) to adjust the covariance matrix Pk/k−1. Finally, the turntable experiment was performed, and the effectiveness and superiority of the proposed method were further verified by compared with other nonlinear methods.