Determining how immune cells functionally interact in the tumor microenvironment and identifying their biological roles and clinical values are critical for understanding cancer progression and developing new therapeutic strategies. Here we introduce TimiGP, a computational method to infer inter-cell functional interaction networks and annotate the corresponding prognostic effect from bulk gene expression and survival statistics data. When applied to metastatic melanoma, TimiGP overcomes the prognostic bias caused by immune co-infiltration and identifies the prognostic value of immune cells consistent with their anti- or pro-tumor roles. It reveals the functional interaction network in which the interaction X→Y indicates a more positive impact of cell X than Y on survival. This network provides immunological insights to facilitate the development of prognostic models, as evidenced by our computational-friendly, biologically interpretable, independently validated models. By leveraging single-cell RNA-seq data for specific immune cell subsets, TimiGP has the flexibility to delineate the tumor microenvironment at different resolutions and is readily applicable to a wide range of cancer types.