This study examined the association between dietary patterns and the development of frailty during 4-, 8-, 12-year follow-up periods in the population-based Taiwan Study. We used the data of an elderly population aged 53 years and over (n = 3486) from four waves of the Taiwan Longitudinal Study on Aging. Frailty was identified by using the modified Fried criteria and the values were summed to derive a frailty score. We applied reduced rank regression to determine dietary patterns, which were divided into tertiles (healthy, general, and unhealthy dietary pattern). We used multinomial logistic regression models to assess the association between dietary patterns and the risk of frailty. The healthy dietary pattern was characterized by a higher intake of antioxidant drinks (tea), energy-rich foods (carbohydrates, e.g., rice, noodles), protein-rich foods (fish, meat, seafood, and eggs), and phytonutrient-rich foods (fruit and dark green vegetables). Compared with the healthy pattern, the unhealthy dietary pattern showed significant cross-sectional, short-term, medium-term, and long-term associations with a higher prevalence of frailty (odds ratios (OR) 2.74; 95% confidence interval (CI) 1.94–3.87, OR 2.55; 95% CI 1.67–3.88, OR 1.66; 95% CI 1.07–2.57, and OR 2.35; 95% CI 1.27–4.34, respectively). Our findings support recommendations to increase the intake of antioxidant drinks, energy-rich foods, protein-rich foods, and phytonutrient-rich foods, which were associated with a non-frail status. This healthy dietary pattern can help prevent frailty over time in elderly people.