Background Few studies have demonstrated the impact of characteristics like age and sex on the association between hand grip strength (HGS) and mild cognitive impairment (MCI). In this cross-sectional study, we aimed to examine the effects of sex and age on the relationship between HGS and MCI. Methods We enrolled older adults age ≥60 years (n = 1009) and measured HGS and MCI in all participants. We analyzed the differences in MCI prevalence among the different variables. The role of sex and age in the association between MCI and HGS was analyzed using binary logistic regression. Results Women had significantly higher prevalence of MCI than men, as did the older group (age ≥70 years) compared with the younger group (age 60–70 years). In men, the low and middle HGS tertiles were significantly associated with MCI. In contrast, only the low tertile of HGS was associated with MCI in women. In the older group, the low tertile of HGS was significantly associated with MCI, which was not observed in the younger group. Conclusions HGS was associated with MCI in older adults, and this association was stronger in men. HGS may be useful for evaluating MCI in older adults.