Colorectal cancer (CRC) is a leading cause of morbidity and mortality in Canada. CRC screening and other factors associated with early-stage disease can improve CRC treatment efficacy and survival. This study examined factors associated with CRC stage at diagnosis among male and female adults using data from a large prospective cohort study in Alberta, Canada. Baseline data were obtained from healthy adults aged 35–69 years participating in Alberta’s Tomorrow Project. Factors associated with CRC stage at diagnosis were evaluated using Partial Proportional Odds models. Analyses were stratified to examine sex-specific associations. A total of 267 participants (128 males and 139 females) developed CRC over the study period. Among participants, 43.0% of males and 43.2% of females were diagnosed with late-stage CRC. Social support, having children, and caffeine intake were predictors of CRC stage at diagnosis among males, while family history of CRC, pregnancy, hysterectomy, menopausal hormone therapy, lifetime number of Pap tests, and household physical activity were predictive of CRC stage at diagnosis among females. These findings highlight the importance of sex differences in susceptibility to advanced CRC diagnosis and can help inform targets for cancer prevention programs to effectively reduce advanced CRC and thus improve survival.