Background: The purpose of this study was to analyze the effects of general characteristics, fatigue, depression, self-efficacy, job stress and interpersonal relationships on the quality of life (QoL) of caregivers in nursing hospitals and use them as basic data for intervention programs to improve the quality of life of caregivers. Methods: The participants in the study were 137 caregivers, aged 52–76, who were actively working in nursing hospitals. Data were collected from caregivers by visiting 9 hospitals in 6 cities, with a questionnaire of fatigue, depression, self-efficacy, job stress, interpersonal relationship, quality of life. Results: Age, marriage, marital satisfaction, education, education experience of QoL, monthly income, perceived economic status, hobby or leisure activity, and number of disease showed differences in the degree of QoL at a statistically significant level. In stage 1, economic status (β = −0.18, p = 0.033) and hobby or leisure activity (β = 0.19, p = 0.025) were influencing factors (F = 4.58, p < 0.001). In stage 2, monthly income (β = −0.19, p = 0.034) and perceived economic status (β = −0.18, p = 0.035) were influencing factors. In stage 3, age (β = −2.80, p = 0.006), perceived economic status (β = −2.41, p = 0.017), self-efficacy (β = 3.19, p = 0.002) and interpersonal relationship (β = 7.12, p < 0.001) were influencing factors which showed 61.5% explanatory power (F = 12.88, p < 0.001). Since the subject’s fatigue, depression, and stress did not affect the quality of life, further research is needed. Conclusions: In order to improve the quality of life of caregivers, it would be necessary to develop interventions for raising their self-efficacy and interpersonal relationship by considering their degree of economic status, hobby or leisure activity, monthly income, and age.