Population aging is escalating in numerous countries worldwide; among them is Taiwan, which will soon become an aged society. Thus, aging successfully is an increasing concern. One of the factors for achieving successful aging (SA) is maintaining high physical function. The purpose of this study was to determine the physical fitness factors associated with SA in Taiwanese older adults (OAs), because these factors are intervenable. Community-dwelling OAs aged more than 65 years and residing in Northern Taiwan were recruited in this study. They received a comprehensive geriatric assessment, which includes sociodemographic data, health conditions and behaviors, activities of daily living (ADL) and instrumental ADL (IADL) function, cognitive and depressive status, and quality of life. Physical fitness tests included the grip strength (GS), 30-second sit-to-stand (30s STS), timed up-and-go (TUG), functional reach (FR), one-leg standing, chair sit-and-reach, and reaction time (drop ruler) tests as well as the 6-minute walk test (6MWT). SA status was defined as follows: complete independence in performing ADL and IADL, satisfactory cognitive status (Mini-Mental State Examination ≥ 24), no depression (Geriatric Depression Scale < 5), and favorable social function (SF subscale ≥ 80 in SF-36). Adjusted multiple logistic regression analyses were performed. Among the total recruited OAs (n = 378), 100 (26.5%) met the aforementioned SA criteria. After adjustment for sociodemographic characteristics and health condition and behaviors, some physical fitness tests, namely GS, 30s STS, 6MWT, TUG, and FR tests, were significantly associated with SA individually, but not in the multivariate model. Among the physical fitness variables tested, cardiopulmonary endurance, mobility, muscle strength, and balance were significantly associated with SA in Taiwanese OAs. Early detection of deterioration in the identified functions and corresponding intervention is essential to ensuring SA.