Tumors strongly depend on their surrounding tumor microenvironment (TME) for growth and progression, since stromal elements are required to generate the optimal conditions for cancer cell proliferation, invasion, and possibly metastasis. Prostate cancer (PCa), though easily curable during primary stages, represents a clinical challenge in advanced stages because of the acquisition of resistance to anti-cancer treatments, especially androgen-deprivation therapies (ADT), which possibly lead to uncurable metastases such as those affecting the bone. An increasing number of studies is giving evidence that prostate TME components, especially cancer-associated fibroblasts (CAFs), which are the most abundant cell type, play a causal role in PCa since the very early disease stages, influencing therapy resistance and metastatic progression. This is highlighted by the prognostic value of the analysis of stromal markers, which may predict disease recurrence and metastasis. However, further investigations on the molecular mechanisms of tumor–stroma interactions are still needed to develop novel therapeutic approaches targeting stromal components. In this review, we report the current knowledge of the characteristics and functions of the stroma in prostate tumorigenesis, including relevant discussion of normal prostate homeostasis, chronic inflammatory conditions, pre-neoplastic lesions, and primary and metastatic tumors. Specifically, we focus on the role of CAFs, to point out their prognostic and therapeutic potential in PCa.