Work ability (WA) reflects an individual’s resources, work demands, and related environment. Self-reports have shown that higher physical activity (PA) is associated with better WA. This study investigated associations of accelerometer-measured (24/7) physical behavior and cardiorespiratory fitness (CRF) with WA. In the FinFit2017-population-based study, the physical behavior of 20–69-year-old working Finns was measured in terms of PA, standing, and sedentariness using validated MAD-APE algorithms based on raw triaxial accelerometer data. During waking hours, the accelerometer was hip-worn, while during the time in bed (TIB), it was worn on the non-dominant wrist. CRF was measured with a 6 min walk test. WA was assessed by four questions excerpted from the Work Ability Index (WAI), called the short WAI (sWAI). Participants (n = 1668, mean age 46.6, SD = 10.9, 57% women) scored on average 23.3 on the sWAI (range 6–27), with a higher value indicating a better WA. More minutes in standing (p = 0.001) and in moderate (p = 0.004) and vigorous PA (p < 0.001) as well as a higher step number (p < 0.001) and better CRF (p < 0.001) were associated with a higher sWAI value. More time spent lying down (p < 0.001) and in high-movement (p < 0.001) and total TIB (p = 0.001) was associated with a lower sWAI. Detailed analysis of 24/7 physical behavior can be utilized in identifying individual-related indicators of WA.