The fundamental reason for the poor performance of recycled aggregate (RA) and recycled aggregate concrete (RAC) is the attached mortar, which severely restricts the application of RA and RAC. This research proposes an innovative method for the modification of RA: using the freeze–thaw cycle (FTC) to remove the attached mortar. In addition, to improve the spalling efficiency, high temperature drying is introduced before and during FTCs. The results show that as the lowest temperature of FTC decreases, the spalling rate increases. Besides, the attached mortar that suffered from high temperature drying is more likely to be peeled off during the following FTCs, and more times of high temperature drying treatments can provide higher efficiency. At the same time, vibrating process can introduce mechanical damage, which is also beneficial for the removal of mortar. Finally, after proper combination of FTC, high temperature drying and vibrating, all attached mortar can be peeled off quickly, and the physical/mechanical properties of modified RA can be very close to the natural aggregates.