Background: Uncertainty exists regarding the associations between physical activity (PA), relative handgrip strength (RHGS), and health-related quality of life (HRQoL) among elderly Korean cancer survivors. This study investigated the moderating effect of RHGS on the relationship between PA and HRQoL in 308 cancer survivors aged ≥65 years using the data from the Korean National Health and Nutrition Examination Survey in 2014–2019. Methods: HRQoL was evaluated with the EuroQol-5-dimension instrument. PA and handgrip strength were assessed with the Global Physical Activity Questionnaire and a hand dynamometer, respectively. Age, sex, body mass index, income, marital status, educational background, heavy alcohol consumption, smoking status, multimorbidity, and hemoglobin were included as covariates. Results: Bivariate logistic regression showed that insufficient PA and no PA were associated with higher odds of a low HRQoL (odds ratio, OR = 2.6, 95% confidence interval, CI = 1.3~5.1, p = 0.005; OR = 2.4, 95% CI = 1.1~5.0, p = 0.024, respectively), compared with sufficient PA (OR = 1). Weak RHGS was associated with high odds of a low HRQoL (OR = 2.6, 95%= 1.530~4.6, p < 0.001) compared with normal RHGS (OR = 1). Particularly, RHGS (β = −0.0573, 95% CI = −0.1033~−0.0112) had a significant moderating effect on the relationship between PA and HRQoL even after adjustments for all the covariates. The negative influence of physical inactivity on HRQoL was observed in cancer survivors with weak RHGS but not in cancer survivors with normal RHGS. Conclusions: The current findings suggest that maintaining or promoting muscular strength through regular exercise is critical for the HRQoL of elderly Korean cancer survivors.