Developing an efficient approach to improve the luminescence of the phosphors without heating processing is a challenge, but attracts much attention. In the present paper, prismatic microcrystals of RE4O(OH)9NO3 (RE = Y, Eu) were synthesized by a hydrothermal reaction at 180 °C for 24 h. The reaction with VO3− did not change the crystal structure of the microcrystals and VO3− substituting for NO3− anions did not take place. However, it contributed to the formation of amorphous particles containing VO3− on the surface of a prism, which is similar to the surface corrosion of a metal, called “surface eroding”. Therefore, surface modification was successfully achieved by eroding the surface of the microcrystals through the reaction with vanadate ions. As a result of VO3−→Eu3+ energy transfer and the light-harvesting ability of VO3−, the red emission intensity at 617 nm of the modified microcrystals greatly increased. Eroding the surface of rare earth microcrystals recommends a new paradigm for luminescence improvement of rare earth compounds.