Prostate cancer is the most common cancer diagnosed in men and the second leading cause of death in male cancer patients. The WHO suggests that cobalt is involved in the carcinogenesis of prostate cancer. There are, however, no studies associating cobalt levels and prostate cancer patient survival. In this study, 261 Polish prostate cancer (n = 261) patients were recruited into a prospective cohort between 2009 and 2015. Serum cobalt levels were measured using ICP-MS after prostate cancer diagnosis and before treatment. All study participants were assigned into quartiles (QI-QIV) based on the distribution of serum cobalt levels among censored patients. Univariable and multivariable COX regression models were used to calculate hazard ratios (HRs) for each serum cobalt level quartile. We found a significant relationship between high serum cobalt levels and poor prostate cancer patient total survival (HR = 2.60; 95% CI: 1.17–5.82; p = 0.02). In relation to prostate cancer patients who died as a result of other non-cancer causes, the association with high levels of cobalt was even stronger (HR = 3.67; 95% CI: 1.03–13.00; p = 0.04). The impact of high serum cobalt levels on overall survival of prostate cancer-specific-related deaths was not statistically significant.