The attitude and heading reference system (AHRS), which consists of tri-axial magnetometer, accelerometer, and gyroscope, has been widely adopted for three-dimensional attitude determination in recent years. It provides an economical means of passive navigation that only relies on gravity and geomagnetic fields. However, despite the advantages of small size, low cost, and low power, the magnetometer and accelerometer are susceptible to external disturbances, such as the magnetic interference from nearby ferromagnetic objects and current-carrying conductors, as well as the motional acceleration of the carrier. To eliminate such disturbances, a vector-based parallel structure is introduced for the attitude filter design, which can avoid the mutual interference between gravity and geomagnetic vectors. Meanwhile, an approach to estimate and compensate the external disturbances in real time for magnetometer and accelerometer is also presented. Compared with existing designs, the proposed filter architecture and external disturbance rejection algorithm can feasibly and effectively cooperate with mainstream data fusion techniques, including complementary filter and Kalman filter. According to experiment results, in the case that large and persistent external disturbances exist, the proposed method can improve the accuracy and robustness of attitude estimation, and it outperforms the existing methods such as switching filter and adaptive filter. Furthermore, through the experiments, the critical role of fading factor in handling the external disturbance is revealed.