This retrospective case-control study evaluated the prevalence of declined swallowing function and the association with oral functions and gender in community-dwelling elderly patients. Their profiles, the results of swallowing function (Eating Assessment Tool: EAT-10) and other oral functions (oral dryness, maximum occlusal force (MOF), tongue–lip motor function (oral diadochokinesis: ODK), maximum tongue pressure (MTP) and masticatory performance (MP)) were extracted for analyses. The patients were categorized into three groups according to EAT-10 score (Group 1: 0, Group 2: 1 and 2, Group 3: ≥3). In total, 242 patients were enrolled and 46 of them (19.0%) were categorized into declined swallowing function (Group 3). In two-group comparisons (Group 1, 2 versus Group 3), significant differences were identified in age and the number of remaining teeth, but they were not identified in three-group comparisons. The patients with declined swallowing function (Group 3) had significantly lower function in ODK and MTP. Multiple logistic regression analyses identified that declined swallowing function was independently associated with declined functions in ODK /ka/ (OR: 5.31, 95% CI: 1.03–27.23, p = 0.04) and in MTP (OR: 2.74, 95% CI: 1.12–6.66, p = 0.03). This study could confirm the critical role of tongue functions in swallowing in community-dwelling elderly patients.