Background Prostate cancer (PC) is estimated to cause 13.1% of all new cancer cases in the United States in 2021. Natural bioactive compounds have drawn the interest of researchers worldwide in their efforts to find novel treatments for PC. Many of these bioactive compounds have been identified from traditional Chinese medicine (TCM) remedies often containing multiple bioactive compounds. However, in vitro studies frequently focus on the compounds in isolation. Aim We used mixture design response surface methodology (MDRSM) to assess changes in PC cell viability after 48 h of treatment to identify the optimal mixture of all 35 three‐compound combinations of seven bioactive compounds from TCM. Methods and Results We used berberine, wogonin, shikonin, curcumin, triptolide, emodin, and silybin to treat PC3 and LNCaP human PC cells at their IC50 concentrations that we calculated. These compounds modulate many chemotherapeutic pathways including intrinsic and extrinsic apoptosis, increasing reactive oxygen species, decreasing metastatic pathways, inhibiting cell cycle progression. We hypothesize that because these compounds bind to unique molecular targets to activate different chemotherapeutic pathways, they will act synergistically to decrease tumor cell viability. Results from MDRSM showed that two‐way combinations were more effective than three‐way or single compounds. Most notably wogonin, silybin, emodin and berberine responded well in two‐compound combinations with each other in PC3 and LNCaP cells. We then conducted cell viability tests combining two bioactive compound ratios with docetaxel (Doc) and found significant results within the LNCaP cell line. In particular, mixtures of berberine and wogonin, berberine and silybin, emodin and berberine, and emodin and silybin reduced LNCaP cell viability up to an average of 90.02%. The two‐compound combinations were significantly better than docetaxel treatment of LNCaP cells. Conclusion Within the PC3 cells, we show that a combination of berberine, wogonin and docetaxel is just as effective as docetaxel alone. Thus, we provide new combination treatments that are highly effective in vitro for treating androgen‐dependent and androgen‐independent PC.