The assistance to older community-dwellers provided by family caregivers is crucial for the maintenance of an acceptable quality of life, especially when dementia is present. The caregiver burden may be extremely high, but few data are available on what patient domains mainly affect the caregiver. The aim of this cross-sectional study, performed in older outpatients, was to examine the impact of cognitive, physical and nutritional status of elderly community-dwellers on the caregiver burden, as evaluated by the Caregiver Burden Inventory (CBI). A group of 406 elderly outpatients (161 M, 245 F, mean age of 83.20 ± 6.40) was enrolled. A significant correlation was observed between Mini Nutritional Assessment Instrument-Short Form (MNA-SF) and CBI (r = −0.34; p < 0.001), suggesting that a poor nutritional status is significantly associated with the caregiver burden. There was also a significant correlation between CBI and Short Physical Performance Battery score (r = −0.29; p < 0.001), hand grip strength (r = −0.25; p < 0.001), Mini-Mental State Examination score (r = −0.39; p < 0.001), Geriatric Depression Scale (r = 0.23; p < 0.001), Body Mass Index (BMI) (r = 0.01; p = 0.03), Activities of Daily Living and Instrumental Activities of Daily Living (ADL/IADL) (r = −0.61 and −0.62, respectively; p < 0.001), and with the 4-m walking speed (r = −0.42; p < 0.001). In the multivariate analysis, only the relationships of the CBI (in particular the physical subcomponent) with ADL, IADL and MNA-SF remained statistically significant (β ± SE −0.89 ± 0.20, p < 0.001; −0.58 ± 0.15, p < 0.001 and −0.25 ± 0.11, p = 0.02, respectively). The relationship between CBI and BMI remained statistically significant only for the physical subcomponent (β ± SE 0.14 ± 0.05; p = 0.006). Thus, in this study, we confirmed that the impairment in the activities of daily living is associated with a significant impact on the caregiver burden, and we found also that a poor nutritional status of the older outpatient is independently more associated with the caregiver burden than cognitive and physical disability. The combined evaluation of both patients and caregivers can improve the knowledge and assistance to the elderly subjects.