Heritability in the immune tumor microenvironment (iTME) has been widely observed, yet remains largely uncharacterized and systematic approaches to discover germline genetic modifiers of the iTME still being established. Here, we developed the first machine learning approach to map iTME modifiers within loci from genome-wide association studies (GWAS) for breast cancer (BrCa) incidence and outcome. A random forest model was trained on a positive set of immune-oncology (I-O) targets using BrCa and immune phenotypes from genetic perturbation studies, comparative genomics, Mendelian genetics, and colocalization with autoimmunity and inflammatory disease risk loci. Compared with random negative sets, an I-O target probability score was assigned to the 1,362 candidate genes in linkage disequilibrium with 155 BrCa GWAS loci. Pathway analysis of the most probable I-O targets revealed significant enrichment in drivers of BrCa and immune biology, including the LSP1 locus associated with BrCa incidence and outcome. Quantitative cell type-specific immunofluorescent imaging of 1,109 BrCa patient biopsies revealed that LSP1 expression is restricted to tumor infiltrating leukocytes and correlated with BrCa patient outcome (HR = 1.73, p < 0.001). The human BrCa patient-based genomic and proteomic evidence, combined with phenotypic evidence that LSP1 is a negative regulator of leukocyte trafficking, prioritized LSP1 as a novel I-O target. Finally, a novel comparative mapping strategy using mouse genetic linkage revealed TLR1 as a plausible therapeutic candidate with strong genomic and phenotypic evidence. Collectively, these data demonstrate a robust and flexible analytical framework for functionally fine-mapping GWAS risk loci to identify the most translatable therapeutic targets for the associated disease.