This study aims to evaluate the differences in body composition, physical function, and physical activity between pre-frail/frail older adults and to detect risk and protective factors against frailty and physical frailty. Fried’s criteria for frailty and physical frailty using the short-performance physical battery (SPPB) were measured in 179 older participants (75.3 ± 6.4 years old). Body weight, height, and waist, arm, and leg circumferences were obtained as body composition variables. Daily accelerometer outcomes (physical activity and inactivity) were obtained. Pre-frail participants showed overall better physical function and spent more time in physical activity and less time in long inactivity periods than frail participants (p < 0.05). Risk frailty factors were higher waist perimeter (Odds Ratio [OR]: 1.032, 95%CI: 1.003–1.062), low leg performance (OR: 1.025, 95%CI: 1.008–1.043), and inactivity periods longer than 30 min (OR:1.002, 95%CI: 1.000–1.005). Protective factors were standing balance (OR:0.908, 95%CI: 0.831–0.992) and SPPB score (OR: 0.908, 95%CI: 0.831–0.992) for frailty, handgrip strength (OR: 0.902, 95%CI: 0.844–0.964) for physical frailty, and light (OR: 0.986, 95%CI: 0.976–0.996) and moderate-to-vigorous (OR: 0.983, 95%CI: 0.972–0.996) physical activity for both. Our findings suggest that handgrip strength, balance, and physical activity are protective frailty factors and can be monitored in pre-frail older adults. Moreover, poor lower body performance and long inactivity periods are frailty risk factors, which highlights their importance in frailty assessment.