YTaO4 and the relevant modification are considered to be a promising new thermal barrier coating. In this article, phase stability and mechanical properties of the monoclinic (M), monoclinic-prime (M′), and tetragonal (T) REMO4 (M = Ta, Nb) are systematically investigated from first-principles calculations method based on density functional theory (DFT). Our calculations show that M′-RETaO4 is the thermodynamically stable phase at low temperatures, but the stable phase is a monoclinic structure for RENbO4. Moreover, the calculated relative energies between M (or M′) and T phases are inversely proportional to the ionic radius of rare earth elements. It means that the phase transformation temperature of M′→T or M→T could decrease along with the increasing ionic radius of RE3+, which is consistent with the experimental results. Besides, our calculations exhibit that adding Nb into the M′-RETaO4 phase could induce phase transformation temperature of M′→M. Elastic coefficient is attained by means of the strain-energy method. According to the Voigt–Reuss–Hill approximation method, bulk modulus, shear modulus, Young’s modulus, and Poisson’s ratio of T, M, and M’ phases are obtained. The B/G criterion proposed by Pugh theory exhibits that T, M, and M’ phases are all ductile. The hardness of REMO4 (M = Ta, Nb) phases are predicted based on semi-empirical equations, which is consistent with the experimental data. Finally, the anisotropic mechanical properties of the REMO4 materials have been analyzed. The emerging understanding provides theoretical guidance for the related materials development.