Physical activity reduces the risk of several noncommunicable diseases, and a number of studies have found self-reported physical activity to be associated with sickness absence. The aim of this study was to examine if cardiorespiratory fitness, device-measured physical activity, and sedentary behaviour were associated with sickness absence among office workers. Participants were recruited from two Swedish companies. Data on sickness absence (frequency and duration) and covariates were collected via questionnaires. Physical activity pattern was assessed using ActiGraph and activPAL, and fitness was estimated from submaximal cycle ergometry. The sample consisted of 159 office workers (67% women, aged 43 ± 8 years). Higher cardiorespiratory fitness was significantly associated with a lower odds ratio (OR) for both sickness absence duration (OR = 0.92, 95% confidence interval (CI) 0.87–0.96) and frequency (OR = 0.93, 95% CI 0.90–0.97). Sedentary time was positively associated with higher odds of sickness absence frequency (OR = 1.03, 95% CI 0.99–1.08). No associations were found for physical activity at any intensity level and sickness absence. Higher sickness absence was found among office workers with low cardiorespiratory fitness and more daily time spent sedentary. In contrast to reports using self-reported physical activity, device-measured physical activity was not associated with sickness absence.