Objective To investigate the prevalence and associated factors of poor sleep quality among community-dwelling elderly population in a rural area of Northern China. Methods We conducted a cross-sectional survey in August-December 2014 and recruited 2195 participants who were aged 65 years or older and living in Yanlou Town of Yanggu County in western Shandong Province, China. Data on demographics, health-related behaviors, and clinical conditions were collected through structured interviews. The Pittsburgh Sleep Quality Index (PSQI) was used to assess the sleep quality and patterns. Poor sleep quality was defined as a PSQI score > 7. We employed multiple logistic models to relate poor sleep quality to various factors. Results The overall prevalence rates of poor sleep quality were 33.8% in the total sample, 39.2% in women and 26.3% in men (P < 0.01). The most common abnormal sleep domains were prolonged sleep latency (39.7%), decreased sleep duration (31.0%), and reduced habitual sleep efficiency (28.8%). Multiple logistic regression analyses revealed that poor sleep quality was significantly associated with female sex (OR = 1.76, 95% CI 1.46-2.12) and clinical comorbidities such as hypertension (OR = 1.28, 95% CI 1.06-1.54), coronary heart disease (OR = 1.60, 95% CI 1.27-2.00), and chronic obstructive pulmonary disease (OR = 1.82, 95% CI 1.34-2.49). Conclusions The sleep disorders were highly prevalent among the elderly in rural China. Modifiable risk factors such as cardiometabolic risk factors and disorders were associated with poor sleep quality, which might be potential targets for interventions to improve sleep quality in elderly population.