The aim of this study was to explore whether drinking herbal tea and tea would positively benefit activities of daily living (ADL) in the elderly. We used data from the Chinese longitudinal healthy longevity survey (CLHLS) to explore the association. Drinking herbal tea and drinking tea were divided into three groups using latent class analysis (LCA): frequently, occasionally, and rarely. ADL disability was measured by the ADL score. Multivariate COX proportional hazards models with competing risks were used to explore the impact of drinking herbal tea and tea on ADL disability, statistically adjusted for a range of potential confounders. A total of 7441 participants (mean age 81.8 years) were included in this study. The proportions of frequently and occasionally drinking herbal tea were 12.0% and 25.7%, respectively. Additionally, 29.6% and 28.2% of participants reported drinking tea, respectively. Multivariate COX regression showed that compared with rarely drinking, frequently drinking herbal tea could effectively reduce the incidence of ADL disability (HR = 0.85, 95% CI = 0.77–0.93, p = 0.005), whereas tea drinking had a relatively weaker effect (HR = 0.92, 95% CI = 0.83–0.99, p = 0.040). Subgroup analysis found that frequently drinking herbal tea was more protective for males under 80 years old (HR = 0.74 and 0.79, respectively), while frequently drinking tea was somewhat protective for women (HR = 0.92). The results indicate that drinking herbal tea and tea may be associated with a lower incidence of ADL disability. However, the risks associated with using Chinese herb plants still deserve attention.