In this study, the effects of annealing on the surface residual impurities and intrinsic defects of unintentionally doped (UID) β-Ga2O3 are investigated by adopting high-temperature thermal treatments at 1000 °C for 1 h under vacuum and O2 ambience. It is found that the recovery between the divacancies VGa+VO and interstitials (Oi) occurs during annealing, and the residual impurities are identified as Si and Cr, which are repelled toward the surface during annealing. Interestingly, these impurities occupy the formation of Ga vacancies (VGa) near the surface formed by oxygen annealing, consequently weakening the relevant impurity scattering and improving carrier mobility. Moreover, the carrier density of the samples is explored using temperature-dependent Hall measurements, which show a slight reduction in both vacuum and oxygen annealing. This reduction might be a result of the VGa pushing the Fermi level away from the conduction band. In addition, the activation energy of Si ions occupying VGa(I) is lower than that of the interstitial Si ions.