Physical activity is inversely associated with risk of some cancers. The relation with cancer‐specific death remains uncertain. Mainly, studies on relationships between physical activity and cancer are based on self‐reported physical activity (SPA). Hereby, we examined whether measured cardiorespiratory fitness (CRF) is associated with cancer risk, mortality, and case fatality. We also describe relationships between SPA and these outcomes, and between CRF and SPA. A cohort of 1997 healthy Norwegian men, aged 40–59 years at inclusion in 1972–75, was followed throughout 2012. At baseline, CRF was objectively measured. SPA (leisure time and occupational) was obtained through a questionnaire. Relationships between CRF or SPA, and the outcomes were estimated using Cox regression, adjusted for age, body mass index (BMI), and smoking. Pearson correlation coefficients evaluated agreements between CRF and SPA. During follow‐up, 758 men were diagnosed with cancer and 433 cancer deaths occurred. Analyses revealed lower cancer risk (Hazard ratio [HR] 0.85, 95% confidence intervals [CI]: 0.68–1.00), mortality (HR 0.68, 95% CI: 0.53–0.88), and case fatality (HR 0.74, 95% CI: 0.57–0.96), in men with high CRF compared to low CRF. Light leisure time SPA was associated with lower cancer risk (HR 0.70, 95% CI: 0.56–0.86) and mortality (HR 0.64 95% CI: 0.49–0.83), whereas strenuous occupational SPA was associated with higher risks (HR 1.42, 95% CI: 1.13–1.78 and HR 1.45, 95% CI: 1.09–1.93). Correlations between CRF and SPA were 0.351 (P < 0.001) and −0.106 (P < 0.001) for leisure time and occupational SPA, respectively. A high midlife CRF may be beneficial for cancer risk, cancer mortality, and case fatality.