Background Change in bowel habit as a sole alarm symptom for colorectal cancer is disputed. Objective We investigated the diagnostic value of change in bowel habit for colorectal cancer, particularly as a single symptom and within different age groups. Methods This retrospective cohort study examined colorectal cancer fast track referrals and outcomes across four Swedish hospitals (April 2016–May 2017). Entry criteria constituted one or more of three alarm features: anaemia, visible rectal bleeding, or change in bowel habit for more than 4 weeks in patients over 40 years of age. Patients were grouped as having only change in bowel habit, change in bowel habit plus anaemia/bleeding or anaemia/bleeding only. Results Of 628 patients, 22% were diagnosed with colorectal cancer. There were no cases of colorectal cancer in the only change in bowel habit group under 55 years, while this was 6% for 55–64 years, 8% for 65–74 years and 14% for 75 years and older. Among subjects under 55 years, 2% with anaemia/bleeding had colorectal cancer, this increased to 34% for 55 years and older ( P < 0.0001). Change in bowel habit plus anaemia/bleeding gave a colorectal cancer prevalence of 16% in under 55 years and increased to 30% for 55 years and older ( P = 0.07). Conclusion Change in bowel habit as the only alarm feature has a low diagnostic yield for colorectal cancer in patients under 55 years.