Our understanding of neuroendocrine prostate cancer (NEPC) has assumed a new perspective in light of the recent advances in research. Although classical NEPC is rarely seen in the clinic, focal neuroendocrine trans-differentiation of prostate adenocarcinoma occurs in about 30% of advanced prostate cancer (PCa) cases, and represents a therapeutic challenge. Even though our knowledge of the mechanisms that mediate neuroendocrine differentiation (NED) is still evolving, the role of androgen deprivation therapy (ADT) as a key driver of this phenomenon is increasingly becoming evident. In this review, we discuss the molecular, cellular, and therapeutic mediators of NED, and emphasize the role of the tumor microenvironment (TME) in orchestrating the phenotype. Understanding the role of the TME in mediating NED could provide us with valuable insights into the plasticity associated with the phenotype, and reveal potential therapeutic targets against this aggressive form of PCa.