The proportion of the world’s population that are over 60 years old is increasing rapidly. The physical and mental health of older people is affected by depression. Health literacy is a major determinant of health and healthcare for the aging; middle-aged and older people with high health literacy are more likely to maintain a healthy lifestyle, and control or manage their chronic diseases. Therefore, this study explored the relationship between health literacy, social support with exchange, and depression, in middle-aged and older adults in the community, using data from the 2015 Taiwan Longitudinal Study on Aging (TLSA) database. Of the 7636 participants, 1481 (19.4%) were middle-aged or older persons with depression symptoms. We found age, gender, and education level to be significantly related to health literacy status, social support with exchange, and depression. Health literacy was positively correlated with depression and social exchange in social support with exchange, whereas the emotional support component of social support with exchange was negatively correlated with depression. Regression-based process analysis was used to verify the mediation effect of health literacy. Our results indicated that when health literacy was entered into the regression model (a × b path), the effect of social exchange on depression was insignificant (c′ = −0.01, p = 0.84), indicating a complete mediation effect. These findings suggest that improving health literacy may offset the impact of social support with exchange on depression, and lead to the mitigation of depression in middle-aged and older people in Taiwanese communities.