An unmanned aerial vehicle (UAV) aeromagnetic measurement system can precisely obtain large-scale data with high efficiency and low cost in regions with complex exploration conditions, which has aroused interest in the field of aerial survey. In this paper, based on the unmanned helicopter platform, an aeronautical magnetic field measurement system is integrated to complete the integration of a rotor UAV, magnetometers, and data recorders. Equipped with a three-axis fluxgate magnetometer for real-time flight compensation, the compensation deviation is ±0.052 nT, which meets the aeromagnetic survey specifications. The UAV can fly autonomously with a maximum load of 45 kg at a speed of 60 km/h for 2 h, and the wind resistance level is level 7. The system has completed a field test of 111.9 km in the Ma'anshan crisis mine region of China and collected 145,495 data points. Then, the adaptive down-sampling method is used to compress the data (to 1.7% of the original data), and it is applied to the three-dimensional magnetic inversion, which successfully obtains the spatial distribution characteristics of the underground orebody, improving the inversion efficiency while ensuring the target area resolution. Six favorable magnetite prospecting sites are delineated in the survey area through inversion results, which demonstrates the potential of the developed UAV aeromagnetic system in the exploration over the areas with complex terrain.