Background: Tumor microenvironment is essential for breast cancer progression and metastasis. Our study sets out to examine the genes affecting stromal and immune infiltration in breast cancer progression and prognosis. Materials and Methods: This work provides an approach for quantifying stromal and immune scores by using ESTIMATE algorithm based on gene expression matrix of breast cancer patients in TCGA database. We found differentially expressed genes (DEGs) through limma R package. Functional enrichments were accessed through Gene Ontology (GO) analysis and Kyoto Encyclopedia of Genes and Genomes (KEGG) pathway analysis. Besides, we constructed a protein-protein network, identified several hub genes in Cytoscape, and discovered functionally similar genes in GeneMANIA. Hub genes were validated with prognostic data by Kaplan-Meier analysis both in The Cancer Genome Atlas (TCGA) database and Molecular Taxonomy of Breast Cancer International Consortium (METABRIC) database and a meta-analysis of hub genes prognosis data was utilized in multiple databases. Furthermore, their relationship with infiltrating immune cells was evaluated by Tumor IMmune Estimation Resource (TIMER) web tool. Cox regression was utilized for overall survival (OS) and recurrence-free survival (RFS) in TCGA database and OS in METABRIC database in order to evaluate the impact of stromal and immune scores on patients prognosis. Results: One thousand and eighty-five breast cancer patients were investigated and 480 differentiated expressed genes (DEGs) were found based on the analysis of mRNA expression profiles. Functional analysis of DEGs revealed their potential functions in immune response and extracellular interaction. Protein-protein interaction network gave evidence of 10 hub genes. Some of the hub genes could be used as predictive markers for patients prognosis. In this study, we found that tumor purity and specific immune cells infiltration varied in response to hub genes expression. The multivariate cox regression highlighted the fact that immune score played a detrimental role in overall survival (HR = 0.45, 95% CI: 0.27-0.74, p = 0.002) and recurrence-free survival (HR = 0.41, 95% CI: 0.22-0.77, p = 0.006) in TCGA database. These result was confirmed in Xu et al. Tumor Microenvironment in Breast Cancer METABRIC database that immune score was a protector of OS (HR = 0.88, 95% CI: 0.77-0.99, p = 0.039). Conclusions: Our findings promote a better understanding of the potential genes behind the regulation of tumor microenvironment and cells infiltration. Immune score should be considered as a prognostic factor for patients' survival.