A digital healthcare system based on augmented reality (AR) has promising uses for postoperative rehabilitation. We compared effectiveness of AR-based and conventional rehabilitation in patients after rotator cuff repair (RCR). This study randomly allocated 115 participants who underwent RCR to digital healthcare rehabilitation group (DR group) and conventional rehabilitation group (CR group). The DR group performed AR-based home exercises using UINCARE Home+, whereas the CR group performed brochure-based home exercises. The primary outcome was a change in the Simple Shoulder Test (SST) score between baseline and 12 weeks postoperatively. The secondary outcomes were the Disabilities of the Arm, Shoulder and Hand (DASH) score; Shoulder Pain And Disability Index (SPADI) score; EuroQoL 5-Dimension 5-Level (EQ5D5L) questionnaire score; pain; range of motion (ROM); muscle strength; and handgrip strength. The outcomes were measured at baseline, and at 6, 12, and 24 weeks postoperatively. The change in SST score between baseline and 12 weeks postoperatively was significantly greater in the DR group than in the CR group (p = 0.025). The SPADI, DASH, and EQ5D5L scores demonstrated group \(\times\) time interactions (p = 0.001, = 0.04, and = 0.016, respectively). However, no significant differences over time were observed between the groups in terms of pain, ROM, muscle strength, and handgrip strength. The outcomes showed significant improvement in both groups (all p < 0.001). No adverse events were reported during the interventions. AR-based rehabilitation significantly improved shoulder function after RCR compared to conventional rehabilitation. Therefore, as an alternative to the conventional rehabilitation, the digital healthcare system is effective for postoperative rehabilitation. Trial registration: ClinicalTrials.gov (identifier: NCT04511377). Registered on August 10, 2020. http://clinicaltrials.gov/ct2/show/NCT04511377