In order to improvethe yaw angle accuracy of multi-rotor unmanned aerial vehicle and meet the requirement of autonomous flight, a new calibration and compensation method for magnetometer based on Levenberg–Marquardt algorithm is proposed in this paper. A novel mathematical calibration model with clear physical meaning is established. “Hard iron” error and “Soft iron” error of magnetometer which affect the yaw accuracy of unmanned aerial vehicle are compensated. Initially, Levenberg–Marquardt algorithm is applied to the process of sphere fitting for the original magnetometer data; the optimal estimation of sphere radius and initial “Hard iron” error are obtained. Then, the ellipsoid fitting is performed, and the optimal estimation of “Hard iron” error and “Soft iron” error are obtained. Finally, the calibration parameters are used to compensate for the magnetometer’s output during unmanned aerial vehicle flight. Traditional ellipsoid fitting based on least squares algorithm is taken as reference to prove the effectiveness of the proposed algorithm. Semi-physical simulation experiment proves that the proposed magnetometer calibration method significantly enhances the accuracy of magnetometer. Static test shows that the yaw angle error is reduced from 1.2° to 0.4° when using the proposed calibration model to calibrate magnetometers. In dynamic tests, the sensor MTi’s output is used as reference. The data fusion of magnetometer compensated by the proposed new calibration model based on Levenberg–Marquardt algorithm can accurately track the desired attitude angle. Experimental results indicate that the accuracy of magnetometer in the yaw angle estimation has been greatly enhanced. In the process of attitude estimated, the compensation magnetometer data given by this new method have faster convergence speed, higher accuracy, and better performance than the compensation magnetometer data given by traditional ellipsoid fitting based on least squares algorithm.