High-temperature annealing has been regarded as an effective technology to improve the performance of Ga2O3-based solar-blind photodetectors (SBPDs). However, as a metastable phase, ε-Ga2O3 thin film may undergo phase transformation during post-annealing. Therefore, it is necessary to investigate the effect of the phase transition and the defect formation or desorption on the performance of photodetectors during post-annealing. In this work, the ε-Ga2O3 thin films were grown on c-plane sapphire with a two-step method, carried out in a metal-organic chemical vapor deposition (MOCVD) system, and the ε-Ga2O3 metal-semiconductor-metal (MSM)-type SBPDs were fabricated. The effects of post-annealing on ε-Ga2O3 MSM SBPDs were investigated. As a metastable phase, ε-Ga2O3 thin film undergoes phase transition when the annealing temperature is higher than 700 °C. As result, the decreased crystal quality makes an SBPD with high dark current and long response time. In contrast, low-temperature annealing at 640 °C, which is the same as the growth temperature, reduces the oxygen-related defects, as confirmed by X-ray photoelectron spectroscopy (XPS) measurement, while the good crystal quality is maintained. The performance of the SBPD with the post-annealing temperature of 640 °C is overall improved greatly compared with the ones fabricated on the other films. It shows the low dark current of 0.069 pA at 10 V, a rejection ratio (Rpeak/R400) of 2.4 × 104 (Rpeak = 230 nm), a higher photo-to-dark current ratio (PDCR) of 3 × 105, and a better time-dependent photoresponse. These results indicate that, while maintaining no phase transition, post-annealing is an effective method to eliminate point defects such as oxygen vacancies in ε-Ga2O3 thin films and improve the performance of SBPDs.