Background: Because dementia is a long-term condition, the appropriate involvement of health-care professionals is considered important. However, little is known about the factors associated with changes in family caregiver burden. Objective: To clarify changes in family caregiver burden and associated factors during follow-up at a memory clinic. Methods: A retrospective cohort study was conducted, enrolling 495 pairs of patients with dementia or mild cognitive impairment and their family caregivers. A total of 120 pairs completed the second evaluation. The caregiver burden was assessed using the Zarit Burden Interview (ZBI). Data at the initial visit and after an average follow-up of about 2 years were compared and analyzed. Results: At initial visit, the patients’ mean age, Mini-Mental State Examination (MMSE) and ZBI scores were 78.6±5.6 years, 23.3±3.5, and 22.6±16.7, respectively. At follow-up, MMSE scores decreased (21.4±4.5, p < 0.001), but ZBI scores remained similar (22.5±13.6). When the difference in ZBI scores between the two time points was defined as ΔZBI, and the related factors were analyzed by multiple regression analysis, ZBI scores at the initial visit, start of psychotropic drug, and decrease of neuropsychiatric symptoms were identified as significant factors (p < 0.001, p = 0.003, p < 0.001, respectively). A significant negative correlation was found between ZBI scores at the initial visit and ΔZBI (r = –0.588, p < 0.001). Conclusions: These findings suggest the importance of assessing changes in the burden experienced by family caregivers during the disease follow-up.