Modifiable factors associated with cognitive decline (CD) require more attention, particularly dietary patterns. This study aimed to investigate the link between cognitive decline and associated factors, particularly dietary patterns (DPs), in community-dwelling older Lebanese of modest economic status. Our cross-sectional national study included 352 participants above 60 years old, from the medico-social centers of the ministry of social affairs all over the country. CD was screened based on literacy. Nutritional and dietary data were collected through a validated food frequency questionnaire. DPs were extracted by the K-mean cluster analysis. CD was found in 32.7% and 61.5% of literate and illiterate groups, respectively. Identified DPs included a Westernized type and Mediterranean type, with high and moderate food intakes. In the context of literacy, independent factors associated with CD were age above 80 years, living in Beirut, frailty, and adopting a Westernized (OR = 3.08, 95% CI: 1.22–7.8) and a high-intake Mediterranean DP (OR = 2.11, 95% CI: 1.05–4.22). In the context of illiteracy, the same factors were associated with CD, but not DP nor frailty, with an age cut-off at 78 years. In a Lebanese sample of older adults, factors associated with CD depend on the level of literacy, with DP only associated with CD in the context of literacy.