The Measurement-While-Drilling (MWD) system, composed of a tri-axial magnetometer and a tri-axial accelerometer, is widely used in the Horizontal Directional Drilling machine in coal mines. This system can provide attitude information of each measuring point in the borehole, which will eventually allow the trajectory of the borehole to be drawn. The attitude information, however, showed a low-level accuracy, due to the sensor’s imperfection and mounting errors. The accuracy worsened when low-cost sensors were employed, as they had higher random noise. Therefore, an exploration of ways to eliminate the sensor imperfection and mounting tolerance as well as to suppress the noise is needed. In this paper, a feasible calibration approach was designed to address these issues. This new approach combined three foundational calibration algorithms, including the ellipsoidal fitting method, the planar fitting method, and the inner product invariance method. The traditional ellipsoidal fitting method and planar fitting method were optimized by using the recursive least square criterion and omitting the steps of sample data acquisition, respectively. In addition, the noise suppression method was involved in our approach to improve the calibration accuracy. The numerical simulation results showed that the number of sampling points decreased significantly, but the accuracy of the azimuthal angle and the pitch angle fully met the engineering requirements. The experimental results showed that the pitch angle error was reduced by less than 0.5°, and the azimuth error was also reduced by less than 2.5°. It should be noted that this new approach could be implemented without the help of other expensive auxiliary equipment.