BackgroundSilibinin, a biologically active compound of milk thistle, has chemopreventive effects on cancer cell lines. Recently it was reported that silibinin inhibited tumor growth through activation of the apoptotic signaling pathway. Although various evidences showed multiple signaling pathways of silibinin in apoptosis, there were no reports to address the clear mechanism of ROS-mediated pathway in prostate cancer PC-3 cells. Several studies suggested that reactive oxygen species (ROS) play an important role in various signaling cascades, but the primary source of ROS was currently unclear.MethodsThe effect of silibinin was investigated on cell growth of prostate cell lines by MTT assay. We examined whether silibinin induced apoptosis through production of ROS using flow cytometry. Expression of apoptosis-, endoplasmic reticulum (ER)-related protein and gene were determined by western blotting and RT-PCR, respectively.ResultsResults showed that silibinin triggered mitochondrial ROS production through NOX4 expression and finally led to induce apoptosis. In addition, mitochondrial ROS caused ER stress through disruption of Ca2+ homeostasis. Co-treatment of ROS inhibitor reduced the silibinin-induced apoptosis through the inhibition of NOX4 expression, resulting in reduction of both Ca2+ level and ER stress response.ConclusionsTaken together, silibinin induced mitochondrial ROS-dependent apoptosis through NOX4, which is associated with disruption of Ca2+ homeostasis and ER stress response. Therefore, the regulation of NOX4, mitochondrial ROS producer, could be a potential target for the treatment of prostate cancer.Electronic supplementary materialThe online version of this article (doi:10.1186/s12885-016-2516-6) contains supplementary material, which is available to authorized users.