This study presents a new real-time calibration algorithm for three-axis magnetometers by combining the recursive least square (RLS) estimation and maximum likelihood (ML) estimation methods. Magnetometers are widely employed to determine the heading information by sensing the magnetic field of earth; however, they are vulnerable to ambient magnetic disturbances. This makes the calibration of a magnetometer inevitable before it is employed. In this paper, first, a complete measurement error model of the magnetometer is studied, and a simplified model is developed. Then, the real-time RLS algorithm is introduced and discussed in detail, and the unbiased optimal ML is utilized to improve the accuracy of the parameter estimation. The proposed algorithm is advantageous in correcting the parameters in real time and simultaneously obtaining unbiased parameter estimation. Finally, the simulation and experimental results demonstrate that both the accuracy and computational speed of the proposed algorithm is better than those of the widely used bath-processing method. Moreover, the proposed calibration method can be adopted for calibrating other three-axis sensors.