Pancreatic ductal adenocarcinoma (PDAC) is a highly lethal malignancy with a five-year survival rate of only 9%. PDAC is characterized by a dense, fibrotic stroma composed of extracellular matrix (ECM) proteins. This desmoplastic stroma is a hallmark of PDAC, representing a significant physical barrier that is immunosuppressive and obstructs penetration of cytotoxic chemotherapy agents into the tumor microenvironment (TME). Additionally, dense ECM promotes hypoxia, making tumor cells refractive to radiation therapy and alters their metabolism, thereby supporting proliferation and survival. In this review, we outline the significant contribution of fibrosis to the pathogenesis of pancreatic cancer, with a focus on the cross talk between immune cells and pancreatic stellate cells that contribute to ECM deposition. We emphasize the cellular mechanisms by which neutrophils and macrophages, specifically, modulate the ECM in favor of PDAC-progression. Furthermore, we investigate how activated stellate cells and ECM influence immune cells and promote immunosuppression in PDAC. Finally, we summarize therapeutic strategies that target the stroma and hinder immune cell promotion of fibrogenesis, which have unfortunately led to mixed results. An enhanced understanding of the complex interactions between the pancreatic tumor ECM and immune cells may uncover novel treatment strategies that are desperately needed for this devastating disease.