Pancreatic cancer remains one of the most lethal malignancies and is becoming a dramatically increasing cause of cancer-related mortality worldwide. Abundant desmoplastic stroma is a histological hallmark of pancreatic ductal adenocarcinoma. Emerging evidence suggests a promising therapeutic effect of several stroma-modifying therapies that target desmoplastic stromal elements in the pancreatic cancer microenvironment. The evidence also unveils multifaceted roles of cancer-associated fibroblasts (CAFs) in manipulating pancreatic cancer progression, immunity, and chemotherapeutic response. Current state-of-the-art technologies, including single-cell transcriptomics and multiplexed tissue imaging techniques, have provided a more profound knowledge of CAF heterogeneity in real-world specimens from pancreatic cancer patients, as well as in genetically engineered mouse models. In this review, we describe recent advances in the understanding of the molecular pathology bases of pancreatic cancer desmoplastic stroma at multilayered levels of heterogeneity, namely, (1) variations in cellular and non-cellular members, including CAF subtypes and extracellular matrix (ECM) proteins; (2) geographical heterogeneity in relation to cell–cell interactions and signaling pathways at niche levels and spatial heterogeneity at locoregional levels or organ levels; and (3) intertumoral stromal heterogeneity at individual levels. This review further discusses the clinicopathological significance of desmoplastic stroma and the potential opportunities for stroma-targeted therapies against this lethal malignancy.