Cancer survivors consider work as a key aspect of cancer survivorship while previous research indicated that cancer survivors have a higher risk of unemployment. The objectives were to assess: (1) whether colorectal cancer survivors less often have paid employment at diagnosis compared to a population-based reference group, (2) whether colorectal cancer survivors with paid work have a higher risk of loss of employment up to 4 years after diagnosis compared to a population-based reference group and (3) which colorectal cancer survivors are at highest risk of loss of paid employment. In a nationwide register-based study, persons diagnosed with colorectal cancer (N = 12,007) as registered in the Netherlands Cancer Registry, were compared on loss of paid employment with a sex and age-matched population-based reference group (N = 48,028) from Statistics Netherlands. Cox regression analyses were conducted. Colorectal cancer survivors had a higher risk of loss of paid employment (HR 1.56 [1.42, 1.71]). Within the group of colorectal cancer survivors, risk of loss of paid employment was lower for older survivors (>60 vs. 45–55) (HR 0.64 [0.51, 0.81]) and higher for those with a more advanced cancer stage (IV vs. I) (HR 1.89 [1.33, 2.70]) and those receiving radiotherapy (HR 1.37 [1.15, 1.63]). Colorectal cancer survivors at high risk of loss of paid employment may benefit from work support interventions as part of cancer survivorship.