The aim of this study was to explore the relationship between socioeconomic status (SES), physical health and the need for long-term care (NLTC) of the Chinese elderly, and further, to provide evidence-based advice for establishing an LTC system in China. A cross-sectional survey was conducted in Shandong Province, China in 2017 by using multi-stage random sampling method. Data were collected from elderly individuals aged 60 years and older by self-designed questionnaires through face-to face interviews. A total of 7070 participants were finally included in the database (40.3% male, 59.7% female). Chi-square test analysis and structural equation modeling (SEM) were conducted to clarify the association between SES, physical health and NLTC among the Chinese elderly men and women in Shandong Province. The results of the SEM analysis showed that physical health exerted a strong and negative effect on the NLTC for both genders, with a slightly stronger effect found among the elderly men. SES was found to be significantly and negatively related to the NLTC among the elderly women, while no statistical significance was found for the association between SES and NLTC for elderly men. A significant and positive association between SES and physical health was observed among the elderly men and women, with a slightly stronger effect among the elderly women. Implications for lowering the NLTC and developing an LTC system were addressed based on the findings above.