The study aimed to investigate the relationship between mobile phone use and cognitive impairment using the data of the Prevention and Intervention on Neurodegenerative Disease for Elderly in China (PINDEC) survey. A total of 21,732 participants aged 60 years and above in China were recruited using a stratified, multi-stage cluster sampling method, providing information on demographics, lifestyle and health-related characteristics, mobile phone use, and cognitive impairment through face-to-face interviews by trained staff according to a standard protocol. All estimates of rates were weighted by sex, age, and living area (rural or urban) in the elderly Chinese population. The rate of mobile phone usage was 65.5% (14.3% for smartphone use). The prevalence of cognitive impairment in non-users of mobile phone, dumbphone users, and smartphone users were 17.8%, 5.0%, and 1.4%, respectively. The odds of having cognitive impairment in users of dumbphone and smartphone were lower than non-users after adjusting for demographics, lifestyle, and health-related factors (adjusted odds ratio (AOR), 0.39, 95% CI 0.35 to 0.45; p < 0.001; AOR, 0.16, 95% CI 0.11 to 0.25; p < 0.001, respectively). Smartphone use in Chinese elderly people was quite low. A strong correlation was found between mobile phone use and better cognitive function; yet longitudinal studies are warranted to explore the causal relationship. Future design of mobile phone-based interventions should consider the feasibility among those in need.