Background: Plant-based diets are associated with various health benefits; however, their impact on physical performance in aging populations remains unclear. Objectives: To investigate the associations between adherence to plant-based diets and physical performance, focusing on their potential protective effects against age-related declines in function. Methods: Data were obtained from men and women aged 40 years or older in the Baltimore Longitudinal Study of Aging (BLSA) (mean ± SD age: 68 ± 13 years at the first dietary visit; n = 1389). Dietary intake was assessed using a food frequency questionnaire (FFQ). Plant-based diets, calculated from 18 food groups, were categorized as overall (PDI), healthful (hPDI), or unhealthful (uPDI), and their tertiles across visits were analyzed. Multivariable linear mixed-effects models were used to examine the association between repeated measurements of three physical performance outcomes—Short Physical Performance Battery (SPPB), grip strength (kg), and gait speed (m/s)—and adherence to each plant-based diet. Results: In fully adjusted models, SPPB and grip strength were significantly associated with both hPDI and uPDI, but not with PDI. For hPDI, the intermediate tertile showed the greatest benefit, with SPPB scores 0.5 points higher (βT2vs.T1 = 0.50, 95% CI: 0.30–0.70, p < 0.001) over the follow-up period. In contrast, for uPDI, a 0.27-point lower SPPB score was seen (βT3vs.T1 = −0.27, 95% CI: −0.48 to −0.07, p = 0.009). Longitudinally, grip strength was positively associated with hPDI (βT3vs.T1 = 1.14, 95% CI: 0.24–2.05, p = 0.0013). Similar results were observed in older adults aged ≥65 years. Conclusions: Adherence to hPDI may benefit lower body function and muscle strength, while uPDI appears to have adverse effects. This suggests that the quality of plant-based foods is essential for maintaining functional well-being in older adults. Further research is needed to confirm these findings, explore underlying mechanisms, and identify strategies to optimize plant-based dietary patterns for aging populations.