Within the realm of aging, the nexus between diet and health has garnered considerable attention. However, only select studies have amalgamated insights into the correlation between plant and animal food consumption and frailty. Our aim was to appraise the connections between the overall plant-based diet index (PDI), healthful plant-based diet index (hPDI), and unhealthful plant-based diet index (uPDI) and frailty in the elderly, utilizing data from the Chinese Longitudinal Healthy Longevity Survey (CLHLS). This cohort study drew upon CLHLS data spanning from 2008 to 2018. The PDI, hPDI, and uPDI were gauged using a simplified food frequency questionnaire (FFQ). A frailty index, encompassing 35 variables across major health domains, was formulated. Cox proportional hazard models were employed to scrutinize the associations between the three plant-based dietary indices and frailty in older adults, including an exploration of gender disparities in these associations. A cohort of 2883 study participants was encompassed, with 1987 (68.9%) observed to be either frail or in the pre-frail stage. The Cox model with penalized spline exhibited linear associations of PDI, hPDI, and uPDI with the frailty index. Following covariate adjustments, it was discerned that older adults situated in the highest quartiles of PDI (HR = 0.86, 95% CI: 0.77–0.95) and hPDI (HR = 0.83, 95% CI: 0.74–0.93) experienced a 14% and 17% diminished risk of frailty compared to those in the lowest quartiles of PDI and hPDI, respectively. Conversely, when contrasted with those in the lowest quartile of uPDI, older adults adhering to the highest tertile of uPDI exhibited a 21% elevated risk of frailty (HR = 1.21, 95% CI: 1.08–1.36), with both associations achieving statistical significance (p < 0.01). Moreover, additional subgroup analyses revealed that the protective effects of PDI and hPDI against frailty and the deleterious effects of uPDI were more conspicuous in men compared to women. To forestall or decelerate the progression of frailty in the elderly, tailored dietary interventions are imperative, particularly targeting male seniors.