Despite the key role played by androgen receptor (AR) in tumor cell aggressiveness and prostate cancer (PCa) progression, its function in the tumor microenvironment (TME) is still controversial. Increasing studies highlight the crucial role played by TME modulation in treatment outcome and tumor cell spreading. In this context, targeting specific constituents of the TME could be considered an alternative approach to classic treatments directed against cancer cells. Currently, androgen deprivation therapy (ADT) is a routinely adopted strategy in the management of PCa, with initial success, and consecutive fail. A possible justification to this is the fact that ADT aims to target all the transcription/translation-related activities of AR, which are typical of tumor epithelial cells. Less is still known about side effects of ADT on TME. Cancer Associated Fibroblasts (CAFs), for example, express a classic AR, mostly confined in the extra-nuclear portion of the cell. In CAFs ADT exerts a plethora of non-transcriptional effects, depending by the protein partner linked to AR, leading to cell migration, proliferation, and differentiation. In recent years, substantial progress in the structure-function relationships of AR, identification of its binding partners and function of protein complexes including AR have improved our knowledge of its signaling axis. Important AR non-genomic effects and lots of its cytoplasmatic binding partners have been described, pointing out a fine control of AR non-genomic pathways. Accordingly, new AR inhibitors have been designed and are currently under investigation. Prompt development of new approaches to target AR or block recruitment of its signaling effectors, or co-activators, is urgently needed. The present review takes an in-depth look at current literature, furnishing an exhaustive state-of-the-art overview of the non-genomic role of AR in PCa, with particular emphasis on its involvement in TME biology.