Limited health literacy (HL) is linked to many negative health outcomes, including poor self-management of chronic diseases and medication adherence among patients. There are a lack of data regarding HL in the elderly population in Malaysia. This study aimed to determine the prevalence of limited HL levels and its associated factors among elderly patients in an urban academic primary care clinic in Selangor, Malaysia. A cross-sectional study was conducted among 413 elderly patients (≥60 years old) who attended this academic primary care clinic between January 2020 and January 2021. Sociodemographic data, clinical characteristics, and health literacy scores were collected. Descriptive statistics (median with interquartile ranges (IQR), frequency, and percentages) and multiple logistic regression were utilized. The prevalence of limited HL in our population was 19.1% (95% CI: 15.3, 23). The middle-old (70–79 years) and very-old (≥80 years) age groups were more likely to have limited HL (aOR 4.05; 95% CI: 2.19, 7.52 and aOR 4.36; 95% CI: 1.02, 18.63, respectively). Those with at least secondary school education (aOR 0.06; 95% CI: 0.02, 0.24) and those who found medical information via the internet/television (aOR 0.21; 95% CI: 0.05, 0.93) had lower odds of having limited HL. In conclusion, having limited HL levels was not common among elderly patients in this primary care clinic. Further studies involving rural and larger primary care clinics in Malaysia are required to support these findings.