Pancreatic ductal adenocarcinoma (PDAC) is one of the deadliest malignancies. It is characterized by a complex and immunosuppressive tumor microenvironment (TME), which is primarily composed of tumor cells, stromal cells, immune cells, and acellular components. The cross-interactions and -regulations among various cell types in the TME have been recognized to profoundly shape the immunosuppression features that meaningfully affect PDAC biology and treatment outcomes. In this review, we first summarize five cellular composition modules by integrating the cellular (sub)types, phenotypes, and functions in PDAC TME. Then we discuss an integrated overview of the cross-module regulations as a determinant of the immunosuppressive TME in PDAC. We also briefly highlight TME-targeted strategies that potentially improve PDAC therapy.