Background: Ambulatory assessments offer opportunities to study physical activity level (PAL) and affect at the group and person-level. We examined bidirectional associations between PAL and affect in a 3-h timeframe and evaluated whether associations differ between people with and without current or remitted depression/anxiety. Methods: Two-week ecological momentary assessment (EMA) and actigraphy data of 359 participants with current (n = 93), remitted (n = 176), or no (n = 90) Composite International Diagnostic Interview depression/anxiety diagnoses were obtained from the Netherlands Study of Depression and Anxiety. Positive affect (PA) and negative affect (NA) were assessed by EMA 5 times per day. Average PAL between EMA assessments were calculated from actigraphy data.Results: At the group-level, higher PAL was associated with subsequent higher PA (b = 0.109, p < .001) and lower NA (b = −0.043, p < .001), while higher PA (b = 0.066, p < .001) and lower NA (b = −0.053, p < .001) were associated with subsequent higher PAL. The association between higher PAL and subsequent lower NA was stronger for current depression/anxiety patients than controls (p = .01). At the person-level, analyses revealed heterogeneity in bidirectional associations.Conclusions: Higher PAL may improve affect, especially among depression/anxiety patients. As the relationships vary at the person-level, ambulatory assessments may help identify who would benefit from behavioral interventions.