The geomagnetic matching aided positioning system based on Iterative Closest Contour Point (ICCP) algorithm can suppress the accumulation error of the inertial navigation system and achieve the accurate positioning of the vehicle. Aiming at the problem that the ICCP algorithm is sensitive to heading error and easily mismatches in regions with similar geomagnetic general features, an improved ICCP matching algorithm based on geomagnetic vector is proposed. The ant colony algorithm is designed to improve the search strategy in a large probability range. The geomagnetic three-dimensional vector feature and the Hausdoff distance are employed as the objective function for multiple iterations, improving matching efficiency and accuracy. Simulation results show that compared with the traditional ICCP algorithm, the positioning error of the matching track, the heading error, and the matching time of the improved ICCP algorithm are reduced by 69.6%, 44.0% and 39.0%, respectively.