Objectives To investigate how changes in physical, social, and cognitive activity levels are associated with the use of the affected upper extremity at different time points during inpatient rehabilitation after stroke. Methods In an observational longitudinal cohort pilot study activity of 14 patients with subacute stroke was registered 2–4 times during their inpatient rehabilitation from 2 p.m. to 8 p.m. At the same time, patients wore accelerometers on both wrists to register amount of use and use ratio of the affected to the unaffected upper extremity. Before and after the observation period, patients were assessed with action research arm test, Box and Blocks Test, and Functional Independence Measure. Linear regression models were used to examine the influence of different categories of activities and motor function levels on affected upper extremity use. Results Increasing physical activity levels during rehabilitation and improvement in upper extremity motor function were associated with increased use of the affected upper extremity. Cognitive and social activity levels did not change and were not associated with affected upper extremity use. Conclusion Our findings suggest that the use of the affected upper extremity and a general increase in activity are associated. Facilitating both general physical activity and specific upper extremity use at a rehabilitation ward may benefit overall recovery.