Background Physical activity may affect disease activity in patients with inflammatory bowel disease. We used a survey to investigate this association and performed interviews to get a better understanding of patient experiences, and therefore the nature of this association. Methods Patients with Crohn’s disease (CD, n = 176) and ulcerative colitis (UC, n = 162) completed the short Crohn’s Disease Activity (sCDAI) or Patient Simple Clinical Colitis Activity Index (P-SCCAI) and the Short Questionnaire to Assess Health-enhancing physical activity (SQUASH). Associations were investigated by multiple linear regression. Semi-structured interviews (7 CD, 7 UC) were conducted to assess patient experiences with the role of physical activity in their disease. Results The majority of survey participants were in remission (70%) and adhered to the Dutch physical activity guidelines (61%). In Crohn’s disease, the total physical activity score was inversely associated with disease activity, even after adjustment for confounders (β = − 0.375; p = 0.013). No association between physical activity and disease activity was found in ulcerative colitis. Of the interviewees, 86% experienced beneficial effects of physical activity, such as improved general fitness, quality of life and self-image. However, during periods of active disease they struggled to find the motivation and perseverance to be physically active due to physical barriers. Conclusions Crohn’s disease participants with a higher physical activity level had a lower disease activity. This inverse association was not found in ulcerative colitis. Interviews revealed that IBD patients generally experience beneficial effects from physical activity, although the barriers caused by active disease may put them off to be physically active.