The strapdown magnetic vector measurement system which can obtain the magnetic vector field and the attitude information simultaneously has widely applied in geophysical prospecting. A high-performance calibration of systematic errors including magnetometer error and misalignment error is essential for this system. The traditional methods calibrate these two errors separately, having the problems of cumbersome steps and be dependent on special data acquisition ways such as rotation, rolling-over, incline, etc. An original method combining plane compression with ellipsoid fitting is proposed in this paper, which can simultaneously complete the calibration calculation of magnetometer error and misalignment error by one experiment. It can be done using a set of attitude-independent spatial scatter point data and no additional mechanical equipment is required. The mathematical analysis of this method has been done to study the elements decreasing the measurement accuracy of the system, and numerical simulation and field experiments are performed to validate the above analysis. The results indicate that the method can do contribution to the accuracy improvement of magnetic vector measurement system.