The accuracy of the airborne vector magnetic (VM) survey mainly depends on the attitude measurement precision of the aircraft. An orientation change of 0.001° can produce an error of approximately 1 nT in VM components for a geomagnetic field of 50,000 nT. Therefore, noise estimation or accurate understanding of noise characteristics of measured airborne VM data plays an important role in geophysical applications. A new airborne VM system with high-precision attitude measurement (0.003°–0.007° root-mean square [rms]) for geophysical prospecting was developed. We have analyzed the results from test flights, compared the difference between measured and calculated VM data, and we developed two methods to estimate the noise in VM data derived from the new system, which include (1) VM noise, that is, the rms error of the difference between the total magnetic intensity calculated from the measured VM data and measured by the scalar sensor, to mainly estimate the error of calibration and compensation, and (2) rms error of the difference between the data from repeat flight lines, to mainly estimate the error of multiple measurements. The noise estimation results of the two methods based on data from test flights in the Qixin area of the East Tianshan Mountains in China indicate that the new system has high accuracy: VM noise is 2.32 nT; the error of the repeat flight lines is 4.86, 6.08, and 2.80 nT for the [Formula: see text], [Formula: see text], and [Formula: see text] components, respectively.