Cancer-associated fibroblasts (CAFs) play key roles in tumor microenvironment; they are thought to originate from adipocytes. This study aimed to evaluate CAF-related protein expression and its implications in breast cancer. Of the 939 enrolled breast cancer patients, 642 had fibrous and 297 had adipose stroma. The status of estrogen receptor, progesterone receptor, human epidermal growth factor receptor 2 (HER-2), Ki-67, podoplanin, prolyl 4-hydroxylase, fibroblast activation protein α (FAPα), S100A4, platelet-derived growth factor receptor α (PDGFRα), PDGFRβ, and chondroitin sulfate proteoglycan (NG2) was evaluated via tissue microarrays. Tumors were divided into luminal A, luminal B, HER-2, or triple-negative breast cancer subtypes according to their molecular status. Luminal A subtype was more prevalent in breast cancer of adipose stroma type, whereas other molecular subtypes were more common in fibrous stroma type (p < 0.001). Tumor cell expression of podoplanin and FAPα was higher in adipose stroma type, while higher expression of prolyl 4-hydroxylase and PDGFRα was observed in fibrous stroma type. Furthermore, adipose stroma type exhibited higher stromal expression of podoplanin, FAPα, PDGFRβ, and NG2, whereas fibrous stroma type had higher prolyl 4-hydroxylase and S100A4 expression. In adipose stroma type, tumor positivity (p = 0.034) and stromal positivity (p = 0.005) of prolyl 4-hydroxylase were associated with shorter disease-free survival, and stromal prolyl 4-hydroxylase positivity was with shorter overall survival (p < 0.001). In conclusion, expression of CAF-related proteins was observed in breast cancer, with different profiles between adipose and fibrous stroma types. Prolyl 4-hydrolase status might be of prognostic value in adipose stroma type.