Advanced prostate cancer (PCa) represents the fifth cause of cancer death worldwide. Although survival has improved with second-generation androgen signaling and Parp inhibitors, the benefits are not long-lasting, and new therapeutic approaches are sorely needed. Lipids and their metabolism have recently reached the spotlight with accumulating evidence for their role as promoters of PCa development, progression, and metastasis. As a result, interest in targeting enzymes/transporters involved in lipid metabolism is rapidly growing. Moreover, the use of lipogenic signatures to predict prognosis and resistance to therapy has been recently explored with promising results. Despite the well-known association between obesity with PCa lethality, the underlying mechanistic role of diet/obesity-derived metabolites has only lately been unveiled. Furthermore, the role of lipids as energy source, building blocks, and signaling molecules in cancer cells has now been revisited and expanded in the context of the tumor microenvironment (TME), which is heavily influenced by the external environment and nutrient availability. Here, we describe how lipids, their enzymes, transporters, and modulators can promote PCa development and progression, and we emphasize the role of lipids in shaping TME. In a therapeutic perspective, we describe the ongoing efforts in targeting lipogenic hubs. Finally, we highlight studies supporting dietary modulation in the adjuvant setting with the purpose of achieving greater efficacy of the standard of care and of synthetic lethality. PCa progression is “a matter of fats”, and the more we understand about the role of lipids as key players in this process, the better we can develop approaches to counteract their tumor promoter activity while preserving their beneficial properties.