The purpose of this study was to investigate the correlations between tumor-to-background ratio (TBR) obtained from breast-specific gamma imaging (BSGI) and the prognostic factors of breast cancer. Sixty-seven patients with invasive ductal carcinoma who underwent preoperative BSGI were enrolled. The BSGI images were visually scored from 1 to 5 according to a breast imaging reporting and data system (BIRADS). The TBR results obtained from positive BSGI images were compared according to the following prognostic factors: tumor size; axillary lymph node metastasis; nuclear grade (NG); histologic grade (HG); subtype; Ki-67; and the expression profile of estrogen receptor (ER), progesterone receptor (PR), and human epidermal growth factor receptor 2 (HER2). Among 67 images, 60 were classified as a positive finding (sensitivity 89.6%). A higher TBR value was significantly correlated with tumor size ≥ 2 cm (P = 0.001), axillary lymph node metastasis (P = 0.007), high HG (P = 0.029), negative PR status (P = 0.036), and Ki-67 ≥ 14% (P = 0.007). The TBR showed a significant difference between the luminal A and non-luminal A subtypes (P = 0.007). On multivariate analysis, TBR had a high correlation with tumor size ≥ 2 cm, axillary lymph node metastasis, and negative PR status (P = 0.003, 0.048, and 0.030, respectively). A high TBR on BSGI was significantly correlated with poor prognostic factors of breast cancer. Luminal A subtype, a breast cancer subtype with more favorable prognosis, was associated with a low TBR on BSGI.