Type 2 diabetes mellitus (T2DM) has consistently been associated with decreased risk of prostate cancer; however, if this decrease is related to the use of anti‐diabetic drugs is unknown. We prospectively studied men in the comparison cohort in the Prostate Cancer data Base Sweden 3.0, with data on T2DM, use of metformin, sulfonylurea and insulin retrieved from national health care registers and demographic databases. Cox proportional hazards regression models were used to compute hazard ratios (HR) and 95% confidence intervals (CI) of prostate cancer, adjusted for confounders. The study consisted of 612,846 men, mean age 72 years (standard deviation; SD = 9 years), out of whom 25,882 men were diagnosed with prostate cancer during follow up, mean time of 5 years (SD = 3 years). Men with more than 1 year's duration of T2DM had a decreased risk of prostate cancer compared to men without T2DM (HR = 0.85, 95% CI = 0.82–0.88) but among men with T2DM, those on metformin had no decrease (HR = 0.96, 95% CI = 0.77–1.19), whereas men on insulin (89%) or sulfonylurea (11%) had a decreased risk (HR = 0.73, 95% CI = 0.55–0.98), compared to men with T2DM not on anti‐diabetic drugs. Men with less than 1 year's duration of T2DM had no decrease in prostate cancer risk (HR = 1.11, 95% CI = 0.95–1.31). Our results gave no support to the hypothesis that metformin protects against prostate cancer as recently proposed. However, our data gave some support to an inverse association between T2DM severity and prostate cancer risk.