Prostate cancer (PCa) cells frequently invade the surrounding stroma, leading to heterogeneous formation of structural atypia. The surrounding stroma contains multiple functionally diverse populations of fibroblasts that trigger numerous changes in PCa cells including motility. Thus, we hypothesized that direct or indirect contact of PCa cells with fibroblasts determines an invasive phenotype in PCa cells. We investigated the effects of 10 different patient‐derived fibroblast lines on the three‐dimensional (3D) morphogenesis of PCa cells growing on a viscous substrate in vitro. When grown alone, all 10 patient‐derived fibroblast lines clumped on the viscous substrate, whereas the human androgen‐sensitive PCa cell line LNCaP did not. Cocultures of LNCaP cells with seven of the patient‐derived fibroblast lines (PrSC, pcPrF‐M5, pcPrF‐M7, pcPrF‐M23, pcPrF‐M24, pcPrF‐M28, and pcPrF‐M31) formed a thick fibroblast layer that resembled human prostate stromal structures. In contrast, cocultures of LNCaP cells with the remaining three fibroblast lines (NPF‐M13, pcPrF‐M10, and pcPrF‐M26) did not form a thick fibroblast layer. Of the seven fibroblast lines that caused thick layer formation, four patient‐derived fibroblast lines (PrSC, pcPrF‐M5, pcPrF‐M28, and pcPrF‐M31) induced an invasive phenotype in LNCaP cells with a cord‐like infiltrating growth pattern, whereas the other three fibroblast lines (pcPrF‐M7, pcPrF‐M23, and pcPrF‐M24) induced no or a very weak invasive phenotype. Using cell culture inserts, none of the four patient‐derived fibroblast lines that induced an invasive phenotype (PrSC, pcPrF‐M5, pcPrF‐M28, and pcPrF‐M31) affected CDH1 mRNA expression in LNCaP cells; yet, two patient‐derived fibroblast lines (pcPrF‐M5 and pcPrF‐M28) increased CDH2 mRNA expression in LNCaP cells, whereas the other two fibroblast lines (PrSC and pcPrF‐M31) did not. These results suggest that the existence of multiple functionally diverse populations of fibroblasts in PCa tissue may be responsible for the diversity in PCa cell invasion, leading to heterogeneous formation of structural atypia.