Pancreatic ductal adenocarcinoma (PDAC) is one of the most aggressive tumors with a poor prognosis. A characteristic of PDAC is the formation of an immunosuppressive tumor microenvironment (TME) that facilitates bypassing of the immune surveillance. The TME consists of a desmoplastic stroma, largely composed of cancer-associated fibroblasts (CAFs), immunosuppressive immune cells, immunoregulatory soluble factors, neural network cells, and endothelial cells with complex interactions. PDAC develops from various precursor lesions such as pancreatic intraepithelial neoplasia (PanIN), intraductal papillary mucinous neoplasms (IPMN), mucinous cystic neoplasms (MCN), and possibly, atypical flat lesions (AFL). In this review, we focus on the composition of the TME in PanINs to reveal detailed insights into the complex restructuring of the TME at early time points in PDAC progression and to explore ways of modifying the TME to slow or even halt tumor progression.