Pancreatic ductal adenocarcinoma (PDAC) remains a lethal cancer. The poor prognosis calls for a more detailed understanding of disease biology in order to pave the way for the development of effective therapies. Typically, the pancreatic tumor is composed of a minority of malignant cells within an excessive tumor microenvironment (TME) consisting of extracellular matrix (ECM), fibroblasts, immune cells, and endothelial cells. Research conducted in recent years has particularly focused on cancer-associated fibroblasts (CAFs) which represent the most prominent cellular component of the desmoplastic stroma. Here, we review the complex crosstalk between CAFs, tumor cells, and other components of the TME, and illustrate how these interactions drive disease progression. We also discuss the emerging field of CAF heterogeneity, their tumor-supportive versus tumor-suppressive capacity, and the consequences for designing stroma-targeted therapies in the future.