Due to the large amount of old hardened cement mortar attached to the surface of aggregate and the internal micro-cracks formed by the crushing process, the water absorption, apparent density, and crushing index of recycled coarse aggregate are still far behind those of natural coarse aggregate. Based on the performance requirements of different qualities of recycled coarse aggregate, the performance differences of recycled coarse aggregate before and after physical strengthening were observed. The results showed that the physical strengthening technique can remove old hardened mortar and micro powder attached to the surface of recycled coarse aggregate by mechanical action, which can effectively improve the quality of recycled coarse aggregate. The optimum calcination temperature of the recycled coarse aggregate was 400 °C and the grinding time was 20 min. The contents of the attached mortar in recycled coarse aggregates of Class I, II, and III were 7.9%, 22.8%, and 39.7%, respectively. The quality of recycled coarse aggregate was closely related to the amount of mortar attached to the surface. The higher the mortar content, the higher the water absorption, lower apparent density, and higher crushing index of the recycled coarse aggregate.