The efficacy of immune checkpoint inhibitors is significantly influenced by the Tumor Immune Microenvironment (TIME). RNA sequencing of tumor biopsies or surgical specimens can offer valuable insights into TIME, but its high cost and long turnaround time seriously restrict its utility in routine clinical examinations. Recent studies have suggested that ultra-high resolution pathological images can infer cellular and molecular characteristics. Motivated by this, we propose a weakly supervised contrastive learning model to deduce tumor microenvironment features from whole slide images (WSIs) of H\&E stained pathological sections. The high-resolution WSIs are split into tiles, and then contrastive learning is applied to extract features of each tile. Following the aggregation of tile-level features, weak supervisory signals are used to fine-tune the encoder. Comprehensive downstream experiments on two independent cohorts and spatial transcriptomics data corroborate that the computational pathological features effectively characterize the proportion of tumor-infiltrating immune cells, immune subtypes, and biomarker gene expression levels. These findings demonstrate that our model can effectively capture pathological features beyond human vision, establishing a mapping relationship between cellular compositions and histological morphology and thereby expanding the clinical practice of digital pathology images.