The purpose of this study was to identify the factors affecting subjective cognitive impairment. We analyzed data from the 2019 Korea Community Health Survey and enrolled 68,546 middle-aged adults, aged 50 to 64 years, and 74,547 older adults, aged 65 years and older, in this study. Multiple logistic regression analysis was performed to identify factors influencing subjective cognitive impairment. Of the participants, 11,926 (17.4%) middle-aged and 21,880 (29.4%) older adults living in the community reported subjective cognitive impairment. Major factors that influenced subjective cognitive impairment in both middle-aged and older adults were gender, subjective stress, depressive symptoms, and alcohol drinking. In contrast to middle-aged adults, the marital status of older adults affected subjective cognitive impairment. Therefore, the factors affecting subjective cognitive impairment in middle-aged and older adults need to be considered for screening and management to prevent cognitive impairment and dementia. In particular, it is necessary to evaluate and manage stress and depressive symptoms from middle age to prevent subjective cognitive impairment.