The “can do, do do” framework combines measures of poor and normal physical capacity (PC, measured by a 6 min walking test, can do/can’t do) and physical activity (PA, measured by accelerometer, do do/don’t do) into four domains and is able to categorize patient subgroups with distinct clinical characteristics, including fall and fracture risk factors. This study aims to explore the association between domain categorization and prospective fall, fracture, and mortality outcomes. This 6-year prospective study included patients visiting a Fracture Liaison Service with a recent fracture. Outcomes were first fall (at 3 years of follow-up, measured by fall diaries), first subsequent fracture, and mortality (at 6 years). Cumulative incidences of all three outcomes were calculated. The association between domain categorization and time to the three outcomes was assessed by uni- and multivariate Cox proportional hazard analysis with the “can do, do do” group as reference. The physical performance of 400 patients with a recent fracture was assessed (mean age: 64 years; 70.8% female), of whom 61.5%, 20.3%, and 4.9% sustained a first fall, sustained a subsequent fracture, or had died. Domain categorization using the “can do, do do” framework was not associated with time to first fall, subsequent fracture, or mortality in the multivariate Cox regression analysis for all groups. “Can’t do, don’t do” group: hazard ratio [HR] for first fall: 0.75 (95% confidence interval [CI]: 0.45–1.23), first fracture HR: 0.58 (95% CI: 0.24–1.41), and mortality HR: 1.19 (95% CI: 0.54–6.95). Categorizing patients into a two-dimensional framework seems inadequate to study complex, multifactorial outcomes. A personalized approach based on known fall and fracture risk factors might be preferable.