One of the key goals of geodesy is to study the fine structure of the Earth’s gravity field and construct a high-resolution gravity field model (GFM). Aiming at the current insufficient resolution problem of the EIGEN_6C4 model, the refined ultra-high degree models EIGEN_3660 and EIGEN_5480 are determined with a spectral expansion approach in this study, which is to augment EIGEN_6C4 model using topographic potential models (TPMs). A comparative spectral evaluation for EIGEN_6C4, EIGEN_3660, and EIGEN_5480 models indicates that the gravity field spectral powers of EIGEN_3660 and EIGEN_5480 models outperform the EIGEN_6C4 model after degree 2000. The augmented models EIGEN_3660 and EIGEN_5480 are verified using the deflection of the vertical (DOV) of China and Colorado, gravity data from Australia and mainland America, and GNSS/leveling in China. The validation results indicate that the accuracy of EIGEN_3660 and EIGEN_5480 models in determining height anomaly, DOV, and gravity anomaly outperform the EIGEN_6C4 model, and the EIGEN_5480 model has optimal accuracy. The accuracy of EIGEN_5480 model in determining south–north component and east–west component of the DOV in China has been improved by about 21.1% and 23.1% compared to the EIGEN_6C4 model, respectively. In the mountainous Colorado, the accuracy of EIGEN_5480 model in determining south–north component and east–west component of the DOV has been improved by about 28.2% and 35.2% compared to EIGEN_6C4 model, respectively. In addition, gravity value comparison results in Australia and mainland America indicate that the accuracy of the EIGEN_5480 model for deriving gravity anomalies is improved by 16.5% and 11.3% compared to the EIGEN_6C4 model, respectively.