Prostate cancer (PCa) lists as the second most lethal cancer for men in western countries, and androgen receptor (AR) plays a central role in its initiation and progression, which prompts the development of androgen deprivation therapy (ADT) as the standard treatment. Prostate tumor microenvironment, consisting of stromal cells and extracellular matrix (ECM), has dynamic interactions with PCa epithelial cells and affects their growth and invasiveness. Studies have shown that both genomic and non-genomic AR signaling pathways are involved in the biological regulation of PCa epithelial cells. In addition, AR signaling in prostate stroma is also involved in PCa carcinogenesis and progression. Loss of AR in PCa stroma is clinically observed as PCa progresses to advanced stage. Especially, downregulation of AR in stromal fibroblasts dysregulates the expression levels of ECM proteins, thus creating a suitable environment for PCa cells to metastasize. Importantly, ADT treatment enhances this reciprocal interaction and predisposes stromal cells to promote cell invasion of PCa cells. During this process, AR in PCa epithelium actively responds to various stimuli derived from the surrounding stromal cells and undergoes enhanced degradation while elevating the expression of certain genes such as MMP9 responsible for cell invasion. AR reduction in epithelial cells also accelerates these cells to differentiate into cancer stem-like cells and neuroendocrine cells, which are AR-negative PCa cells and inherently resistant to ADT treatments. Overall, understanding of the cross talk between tumor microenvironment and PCa at the molecular level may assist the development of novel therapeutic strategies against this disease. This review will provide a snapshot of AR’s action when the interaction of stromal cells and PCa cells occurs.