The stroma-rich, immunosuppressive microenvironment is a hallmark of pancreatic ductal adenocarcinoma (PDA). Tumor cells and other cellular components of the tumor microenvironment, such as cancer associated fibroblasts, CD4+ T cells and myeloid cells, are linked by a web of interactions. Their crosstalk not only results in immune evasion of PDA, but also contributes to pancreatic cancer cell plasticity, invasiveness, metastasis, chemo-resistance, immunotherapy-resistance and radiotherapy-resistance. In this review, we characterize several prevalent populations of stromal cells in the PDA microenvironment and describe how the crosstalk among them drives and maintains immune suppression. We also summarize therapeutic approaches to target the stroma. With a better understanding of the complex cellular and molecular networks in PDA, strategies aimed at sensitizing PDA to chemotherapy or immunotherapy through re-programing the tumor microenvironment can be designed, and in turn lead to improved clinical treatment for pancreatic cancer patients.