Promoting a healthy diet of the elderly is an important task in the current “Healthy China Action”. This study aimed to describe the changing trends of the dietary knowledge elderly Chinese during 2004–2015 and to examine the associated factors of dietary knowledge. Elderly people aged ≥60 years were included as study subjects from the China Health and Nutrition Survey 2004–2015. A total of 15,607 samples were involved in the analysis. The correct rate of dietary knowledge items followed upward trends over time, except for two items regarding physical activity intensity (Question 11, Cochran-Armitage c 2= 20.05, p <0.001) and healthy weight (Question 12, Cochran-Armitage c 2= 43.93, p <0.001). Four of the twelve dietary knowledge items consistently followed the lowest correct rate between 2006 and 2015, regarding physical activity intensity (Question 11, 24.5%−25.8%), staple food consumption (Question 5, 36.6%−41.5%), animal product consumption (Question 6, 45.8%−59.5%), and fatty meat and animal fat consumption (Question 7, 63.6%−64.9%). Participants who had a lower educational level or lived in rural areas or western regions, did not know about the Chinese Food Pagoda (CFP) or Dietary Guidelines for Chinese Residents (DGCR), and did not proactively look for nutrition knowledge were less likely to have adequate dietary knowledge literacy. Targeted interventions should be developed to promote dietary knowledge level of the elderly.